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PREFACE 



Over the last decade, interest in early intervention programs at the secondary school 
level, such as Talent Search, has grown by leaps and bounds among policy makers and educators. 
Both have recognized that early awareness and intervention programs play an important role in 
helping economically disadvantaged students to stay in school, to make the most of their school 
experience, and to pursue further education after high school. 

The Talent Search program offers young people academic, financial and personal 
counseling; tutoring; career exploration; information on postsecondary education and financial aid; 
and many other services. Operating through institutions of higher education, public and private 
community organizations, and public schools, Talent Search touches the lives of over 285,000 
students a year. 

The U.S. Department of Education has not launched an evaluation of the Talent 
Search program in almost 20 years. However, evaluations of the two other original TRIO 
programs for disadvantaged students-Student Support Services and Upwsrd Bound-are currently 
underway. The Department of Education plans to conduct an evaluation of Talent Search 
beginning in Fiscal Year 1994. 

In order to assist staff at the Department of Education in thinking about the 
important questions this evaluation should address and the research designs and methods that 
should be employed, the Department's Office of Policy and Planning (OPP) commissioned six 
papers that would examine these issues. The general purpose of the papers was to help 
Department of Education staff address the following questions: 

■ Which policy questions are most important to attempt to answer through an 
evaluation of Talent Search? 

■ Which program impacts are most important to examine, and which are possible 
to measure through an evaluation? 

■ Which research methods would make sense to use in evaluating Talent Search 
given the nature of the program and the questions that the Department of 
Education might want to answer through an evaluation? 
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The papers, prepared over the summer of 1992, formed the basis of the Design 
Conference for the Evaluation of Talent Search, which was convened on September 30, 1992. The 
authors and other participants discussed the papers and offered a wide range of ideas as to how 
the program should be evaluated. 

This report is a compilation of the commissioned papers, preceded by a synthesis of 
the major themes that emerged during the conference. 
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DESIGN CONFERENCE FOR THE EVALUATION OF TALENT SEARCH 

Synthesis of Major Themes 

Ellen Tenenbaum 
Westat, Inc. 

Introduction 

The Talent Search program originated in the Higher Education Act of 1965 to 
identify financially needy students and help them take advantage of the newly authorized 
Educational Opportunity Grant Program. As a discretionary grant program based at colleges and 
universities and nonprofit agencies, Talent Search was designed to: (a) identify youths of extreme 
financial or cultural need with an "exceptional potential" for postsecondary education and 
encourage them to complete secondary school and undertake further education; (b) publicize 
availability of student financial aid; and (c) encourage secondary school or college dropouts to 
reenter educational programs. 

Administered by the Office of Postsecondary Education of the U.S. Department of 
Education, Talent Search is one of several federally funded college-based programs for 
disadvantaged students collectively known as the TRIO Programs. In academic year 1991-92, 
Talent Search funding of $59,568,000 was awarded to 295 postsecondary institutions and nonprofit 
agencies to serve a total of over 285,000 persons. Individual Talent Search projects range in size, 
with most serving between 500 and 1,000 participants. 

Talent Search projects supplement the normal secondary school precollege counseling 
services. Project-based Talent Search counselors work through the schools, providing students 
from 6th- 12th grade with a variety of services: 

■ Academic, financial, or personal counseling 

■ Career exploration and aptitude assessment 

■ Assistance with the re-entry process to high school or college 

■ Information on postsecondary education 

■ Information on student financial assistance 



■ Assistance in completing college admissions testing, college admissions 
applications, and financial aid applications 

■ Special activities for 6th through 8th graders 

The first national evaluation of Talent Search was conducted by Research Triangle 
Institute (RTI) in the early 1970s. (J.N. Pyecha and others, 1975: A Study of the National Upward 
Bound and Talent Search Programs. Report to the U.S. Department of Education. Washington, 
D.C.: U.S. Office of Education.) The evaluation currently being planned for 1994 will be the first 
large-scale evaluation of Talent Search since that time. In addition to this evaluation, evaluations 
of the Student Support Services and Upward Bound programs, also under TRIO, are in progress. 
The focus of all these evaluations is on improvement - improvement of federal policy, 
improvement of federal management, and improvement of program practices. 

In light of the technical challenges that the evaluation of Talent Search presents, 
experts were commissioned to prepare six papers on topics pertinent to evaluation of the Talent 
Search program. The authors discussed their papers and related issues at a conference convened 
by the U.S. Department of Education on September 30, 1992. This paper synthesizes the major 
themes that emerged from the papers and the discussions that took place at the conference. 

Given that there has been so little previous research on Talent Search, the 
Department of Education's objective was to obtain a wide array of advice from experts. The 
conference did not strive for consensus. The purpose of this paper, accordingly, is to reflect the 
variety of views that were expressed as to how the evaluation of Talent Search might be designed. 
It should not be regarded as constituting a complete or coherent evaluation design. 

The authors, their affiliations, and the titles of their papers are listed below. Where 
similar titles are shown, the intent was to assign the same general topic area to different authors 
and thus gain a variety of views. 



Thomas A. Angelo 

Director, Academic Development Center 
Boston College 

"Designing a Useful and Appropriate Evaluation of Talent Search: What are the 
Most Important Design Questions to Consider?" 



Alvia Y. Branch 

Vice President, Public/Private Ventures 
Philadelphia, Pennsylvania 

Talent Search: Issues for an Evaluation" 



Ann Coles 
Executive Director 

Higher Education Information Center 
Boston, Massachusetts 

"Perspectives on an Evaluation of Talent Search: Interviews with Talent Search Staff 

Amaury Nora Alberto F. Cabrera 

Associate Professor of Higher Education Assistant Professor 

University of Illinois at Chicago Department of Educational 

Administration and Policy Studies 
State University of New York-Albany 

"Measuring Program Outcomes: What Impacts are Important to Assess, and What 
Impacts are Possible to Measure?" 

James E. Rosenbaum 

Professor of Sociology, Education and Social Policy 
Northwestern University 

"Review of Two Studies of Talent Search: The Research Triangle Institute's 1975 
Study of Talent Search and Paul Franklin's 1985 Study for the College Board" 

William T. Trent 

Associate Professor of Educational Policy Studies and Sociology 
University of Illinois at Urbana-Champaign 

"Measuring Program Impact: What Impacts are Important to Assess, and What 
Impacts are Possible to Measure? A Proposal for Research" 



In addition to the authors and the Office of Policy and Planning, other Department of 
Education offices and the National Council of Educational Opportunity Associations were 
represented. The complete list of conference participants follows the set of papers in this volume. 

This paper summarizes the authors' observations and suggestions on five major 
themes that emerged from the conference. The themes can be characterized as follows: 

■ Objectives of the Talent Search program, past and present 

■ Objectives of the evaluation 

■ Identifying and measuring Talent Search impacts 

■ Other methodological issues 

■ Time frame for the next evaluation 

Objectives of the Talent Search Program 

Talent Search is a program in flux. Talent Search originally was intended to serve a 
small number of students with exceptional potential-- 11th and 12th grade students who needed 
only a little information and counseling to ensure their successful pursuit of higher education-and 
this could be provided at little cost. Over the years, however, the program opened its doors not 
only to students with exceptional potential but to a much broader range of students with a larger 
set of needs. Now that it must also serve intermediate-level students, one must ask about the 
relationship between today's target populations and the resources that are brought to bear. 

The Talent Search program was established in the mid-1960s as part of the Equal 
Opportunity Grant (EOG) legislation. EOG grants were to go to exceptionally needy, 
exceptionally talented students. It emerged as part of the civil rights movement to provide access 
to college for black students. The colleges said they did not know where the exceptionally needy, 
exceptionally talented black students were. The job of Talent Search, then, was to search out these 
students and provide them with EOG grants. 

The target population for Talent Search has changed-and greatly expanded-since the 
program's origins in the mid-sixties, and the program's resources may not have caught up with the 
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needs. Talent Search in the 1960s was directed toward those minority students whose grades may 
have been high but who, for lack of money and because of racial issues, were denied access to 
college. Over the years, however, the group of target students greatly expanded to include those 
with more complex needs that may have hindered their ability to do well academically, to include 
increasing numbers of students who have fallen through the cracks at school and to include more 
students who are poor and meet the eligibility requirements. Talent Search is. in a sense, a 
different program from what it was originally, and the consensus of the conference participants 
was that a realistic evaluation of it should take these changes into account. 

Although financial factors were initially seen as the primary obstacle to college 
enrollment for disadvantaged, largely minority youngsters, this view has been superseded by one 
that also emphasizes cultural, academic, and informational barriers. As a result, under the 1992 
Amendments of the Higher Education Act, Talent Search projects may include early intervention 
programs for eligible students in grades 6 through 8. Talent Search at this level is to provide a 
wide range of services, that may include academic tutoring, course planning, instruction in study 
skills and test taking, parent involvement, college orientation, and follow-up into high school. In 
addition, new types of coordination between Talent Search projects and non-federal programs with 
similar objectives are also being encouraged. These developments reflect the changing nature of 
Talent Search and the possibilities for proposing future directions for the program. Still, Talent 
Search was and remains a relatively low- intensity program-both on its own and when viewed 
among the full set of TRIO programs and services. 

Issues io Address Through Evaluation 

Participants agreed that the evaluation's overall effort should match the kind of 
program that Talent Search is today. In Branch s view, evaluators tend to take a modest program 
and use such "heavy artillery" methods as multi-site random assignment and long-term impact 
assessment with econometric modeling to evaluate it. What is needed is a realistic perspective that 
does not expect to find long-term outcomes fsuch as higher college graduation rates) from a low- 
intensity program such as Talent Search. 

At the same time, since Talent Search is not a high-budget or high-intensity program, 
it cannot be evaluated as isolated from other experiences. Its relationships to the school's normal 
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counseling services, to Upward Bound, and to other programs operating at the school are complex. 
In an evaluation, Talent Search must be assessed in the context of these other services to which 
participants may be exposed. The objectives and design of the Talent Search evaluation must 
therefore be multifaceted. 

The questions that authors suggested be addressed in this evaluation could fall into 
four major groupings: 

■ Talent Search and the student 

■ Talent Search and the school 

■ Talent Search and the community 

■ Effective practices of Talent Search projects 

Talent Search and the Student 

1. What services do Talent Search students receive? How intensive are its 
services, and how appropriate are they given the needs of the student? What 
services do Talent Search students receive that they would not otherwise 
receive, or receive to the same degree? 

2. Defining who is a Talent Search participant is an evaluation issue. How many 
contacts does it take to record a student as a Talent Search participant? This 
varies by site evidently. Moreover, students receiving Talent Search services 
and counseling often do not associate the service or counselor specifically with 
Talent Search." They may know it by another name, or they may not know 
they are in a discrete program called Talent Search. 

3. Does Talent Search increase the likelihood that participants will go on to 
college, compared with similar students not served by Talent Search? 
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4. 



Does Talent Search increase the likelihood that participants will graduate high 
school? The main goal of some Talent Search projects is to encourage students 
who would potentially drop out to continue on and earn a high school diploma. 



5. Do Talent Search participants going to college receive adequate financial aid to 
meet their expenses and better financial aid packages than if they had not been 
assisted? 

6. Are Talent Search participants more knowledgeable about college and 
financial aid application processes than they would have been without Talent 
Search? Rosenbaum elaborated that Talent Search has assumed that low- 
income youths need information, and that publicizing information about 
colleges and financial aid can help them attend postsecondary schools. Given 
the complications involved in making these applications, this is certainly true, 
he said. But what it also suggests is that we need to gather clear information 
about exactly what needs to be conveyed to students. This would be extremely 
helpful to Talent Search projects; quite possibly different projects have found 
different ways to do this well. This needs to be studied and disseminated, 
Rosenbaum suggested. 

7. For those projects that emphasize academic services, is the academic 
performance of Talent Search students better than if they had not 
participated? Project directors interviewed by Coles believed that for the 
intensively academic middle-school Talent Search projects this was an 
important question. At the high school level, however, if Talent Search does 
not provide academic services, then Talent Search cannot be held accountable 
for raising academic performance. 

8. Are Talent Search students 9 college retention rates higher than Aose of similar 
students who were not served? While everyone in Coles' survey wanted to 
learn about participants' college retention patterns, about half of them believed 
that given the limited resources and limited services of Talent Search, Talent 
Search cannot be held accountable for students c e they enter college. 
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Rosenbaum's paper reanalyzed RTTs reported success rates in postsecondary 
institutions; he suggested that there are possible differences ir success rates for 
different types of postsecondary institutions. The next evaluation should be 
sensitive to different types of placements. 

Trent, a Talent Search director in the early 1970s, preferred to think of Talent 
Search as an information delivery system and, as such, a program with relatively 
short-term goals: (1) an effective information delivery system, and (2) effective 
assistance to students in completing and submitting their college applications 
and financial aid forms. An evaluation of Talent Search, he believed, should 
measure the extent to which these goals are met. 

The goal of increasing participants' postsecondary enrollment, he felt, should 
not be a fundamental criterion for evaluating the effectiveness of the program; 
while postsecondary enrollment is an appropriate measurable outcome, Talent 
Search should not be held accountable-in terms of funding renewal or as a 
policy requirement— for achieving high rates of postsecondary enrollment, 
retention, or graduation. 

What are the effects of Talent Search on students' aspirations and motivation- 
exerted directly, and through family involvement? 

Are the program's objectives for students too ambitious? Branch said that 
programs tend to seek to achieve outcomes, through an injection of 
interventions, that are too ambitious and do not take into account the complex, 
entrenched problems of the youths' lives. There is a basic mismatch between 
the nature of their problems and the outcomes the program expects from a one- 
time intervention. What is needed, Branch suggested, is a series of 
interventions that, while they cannot match the seriousness of the problems, 
would attempt realistically to meet them through early intervention and 
continuity of care over a several-year period. To what extent do Talent Search 
programs provide a reasonable match? 
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Talent Search and the School 

1. On a general policy level, what are the gaps in the schools' core counseling 
services that seem to require Talent Search or similar external programs? 

According to Rosenbaum, Congress, in designing Talent Search, decided that 
low-income youth were not being adequately served by the public schools, and 
indications are that assumption is correct. There is no systematic information 
about what precollege counseling services are being provk* / the public 
schools. Talent Search raises the policy question of what se^ices are being 
provided by the high schools, and why high schools don't, as a matter of course, 
provide the additional services that Talent Search projects are providing. One 
must ask why schools have difficulty helping these students. This evaluation will 
not be able to address this point completely, but it should consider what public 
schools are doing in these areas where Talent Search projects are operating. 

2. On a more specific level, what is the relationship between Talent Search and 
the school, and how can it be improved? One can look at Talent Search within 
a school in the context of other programs within that school to understand the 
whole. How does Talent Search work relative to the efforts of the school as a 
whole to improve its capacity to prepare students better for graduation and 
college? Is Talent Search perceived as a program that complements or 
enhances school efforts, duplicates existing efforts, enables schools to direct 
their efforts elsewhere, or uproots the school's own precollege counseling role? 

The ways in which different Talent Search projects relate to their schools were 
discussed. For example, a project could marshal its limited resources to 
enhance a school's entire core precollege counseling services; or a project could 
use its resources to serve Talent Search students individually. A school might 
welcome Talent Search counselors or see them as a threat. The relationship 
between Talent Search counselors and the regular school counselors should be 
described and analyzed, several conference participants believed. To what 
extent does Talent Search supplement or substitute for the school's core 
services, and what are the implications of each role? 



Talent Search and the Community 



1. What is the relationship between Talent Search and the postsecondary 
institutions in its service area, and how can it be improved? 

2. What is the relationship between Talent Search and the community 
organizations to which Talent Search counselors refer students, and how can 
they be better coordinated? 

^. What community organizations contribute funds or volunteers to Talent Search, 
and how can such relationships be encouraged? 

4. To what extent does student participation in Talent Search affect the 
motivation of siblings, relatives, or friends to seek out Talent Search or consider 
pursuing higher education? 



Effective Practices of Talent Search Projects 

A major objective of this evaluation, participants agreed, must be to produce 
information and results that are directly useful to Talent Search directors and 
staff for improvement of their programs. Effective practices should be 
identified and described that will improve projects' methods of recruiting and 
selecting students, keeping records, sharing information with students, helping 
them apply for admission and financial aid, targeting services to the age level of 
the students, coordinating with schools and postsecondary institutions, and 
maximizing the use of resources. 

Finally, a crosscutting goal of the Talent Search evaluation--to be applied to all of the 
objectives-is to discover the factors contributing to success of Talent Search programs in their own 
settings, relative to the goals each project has set for itself, and the particular needs of its students. 
Angelo elaborated on this point by suggesting that the next evaluation should define success in 
several different ways, acknowledging that success will mean different things in different Talent 
Search settings. Further, being successful at informing students should be gauged differently from 
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being successful at motivating students. In short, the evaluation, according to conference 
participants, should disaggregate outcomes such as "success"- to learn not just "What works?" but 
what works for whom, when, and under what circumstances. 

Identifying and Measuring Talent Search Impacts 

There has not been a study of the net program impact of Talent Search. Moreover, 
the RTT report and a subsequent College Board report concluded that it is not possible to measure 
net program impacts on student participants in a low-intensity program such as Talent Search. 
(P.L. Franklin, 1985: Helping Disadvantaged Youth and Adults Enter College: An Assessment of 
Two Federal Programs. Washington, D.C: College Entrance Examination Board.) The meeting 
opened up questions as to the possibility of assessing program impacts, whether by attempting to 
measure net program impacts or by assessing program impacts through other strategies, and 
whether a nationally representative study is necessary to address impacts, or whether more limited 
approaches can be effectively employed. 

Short-term versus long-term impacts on participants. Several conference 
participants argued that because Talent Search cannot be held responsible for student outcomes in 
the postsecondary years, the evaluation should not be measuring net impact in terms of college 
retention or graduation rates of former Talent search participants. By the same token, however, 
the Talent Search directors Coles interviewed want very much to know the patterns of college 
progression among students who were in Talent Search and went on to college. Moreover, 
apparently many Talent Search programs maintain follow-up data on their students, making it 
possible to perform some kind of postsecondary analysis. 

Most participants believed that the evaluation can and should measure college 
enrollment rates of Talent Search participants, as well as high school graduation rates. Trent, 
however, suggested that the impacts of Talent Search should be viewed more narrowly and 
measured primarily in terms of whether Talent Search students were adequately informed, 
adequately assisted with forms, and adequately counseled by the Talent Search program. 
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Quantitative or qualitative indicators of program impact. It was generally believed 
that a rigorous, national quantitative analysis alone would be costly, would be difficult to do in 
terms of identifying appropriate student comparison groups, and might find very small net 
program impacts. Moreover, many of the evaluation's objectives could be examined through 
qualitative analysis of program impacts, such as site visits at the project level and the school level 
In this vein, the next evaluation will need to incorporate both quantitative and qualitative 
indicators of program impact. 

Assessing Talent Search outcomes at each stage of the college choice process. The 

paper submitted by Nora and Cabrera suggested that a young person's needs differ at various ages 
and grade levels, that Talent Search services should be appropriately tailored, and that program 
impacts should be assessed at each stage accordingly. Four stages were suggested by the co- 
authors as calling for a tailored set of Talent Search services: 

1. In the predisposition stage, which begins as early as 7th grade, occupational and 
educational aspirations are first developed, and parents \ave a major influence. 
Program impacts would assess students' academic skills, aspirations, and 
whether they enrolled in a college preparatory curriculum. The co-authors 
suggested that Talent Search programs at this stage include (1) career 
exploration and decision-making workshops; (2) academic tutoring; (3) visits to 
colleges and workplaces; (4) assessing academic potential through standardized 
tests; and (5) self-esteem and self-concept workshops. 

2. In the search phase, spanning 10th through 11th grades, the student develops 
firm aspirations to pursue college and accumulates information on higher 
education options. Visiting campuses, obtaining catalogs, and talking with 
friends about college are some of the activities of students in this phase. 
Program impacts would assess the extent to which students arrived at a 
narrowed list of possible postsecondary institutions, and obtained information 
on them. Examples of possible Talent Search services at this stage include (1) 
counseling to help students narrowing their postsecondary choices; (2) 
collecting and disseminating information on the variety of postsecondary 
educational opportunities; (3) providing academic tutorial sessions; and (4) 
participating in "College Days" activities. 
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3. The choice phase, from the end of 11th grade through 12th grade, is marked by 
a matching process between the student's academic strengths, ability to meet 
costs, proximity of the po$tsecondary institution, and other factors leading to a 
final selection of schools. Some program impacts at this stage include student 
awareness of costs, financial aid, and admission standards, as well as actual 
submission of applications. Some examples of appropriate Talent Search 
intervention strategies in the choice phase are (1) counseling assistance in 
filling out admissions and financial aid forms; (2) collecting and disseminating 
information on student financial aid and academic assistance; (3) tutoring and 
academic summer programs; and (4) college orientation workshops. 

4. The process concludes with the enrollment stage, characterized by final 
selection and then enrollment and attendance at a postsecondary institution. 
The size of the financial aid package has a critical influence at this stage. 
Program impacts to assess would include the incidence of pre-registration, 
applying for financial aid, enrollment, and attendance. Talent Search services 
for this phase should include (1) counseling in filling out admissions, financial 
aid, and application forms; and (2) following up on the status of admission, 
financial aid, and actual enrollment of students. 

Other Methodological Issues 

Appropriateness of a single national evaluation. Given that Talent Search takes 
many different forms and serves students of different age groups, several conference participants 
thought that a single national evaluation may not be the recommended approach to take in 
evaluating this program. While Talent Search will be described nationally as a whole, describing 
the variety of types of Talent Search programs and settings must constitute a major component of 
the evaluation. In this vein, a case study component warrants consideration. Angelo suggested 
that before determining the design of the full-fledged evaluation, a few demonstration evaluations 
might be conducted at certain sites. 

Comparison groups. One challenge in designing an evaluation of Talent Search is 
how to identify appropriate student comparison groups in an attempt to compare impacts of 




Talent Search on participants versus nonparticipants. Among possibilities considered were to 
compare participants and nonparticipants (having similar characteristics) within a single school; or 
to compare Talent Search participants in Talent Search schools with students in non-Talent Search 
schools that have characteristics similar to those of the Talent Search schools. Under the latter 
idea, the school may be the unit of analysis, given that Talent Search affects the school it serves. 
One could compare graduation rates or college enrollment rates for schools that have Talent 
Search with those of demographically similar schools that do not have Talent Search. Again, 
though, the differing goals of individual Talent Search programs would have to be taken into 
account in such a comparative study. Moreover, many high schools have several intervention 
programs-not only Talent Search-making comparability more difficult to attain. 

An alternative kind of comparison-apart from one that attempts to measure overall 
net impact by utilizing comparison groups of students— is to compare the goals, services, practices, 
and outcomes across Talent Search programs. For example, in accounting for the 60-90 percent 
spread in college placement rates among the Talent Search directors Coles interviewed, one can 
discover factors that underlie those differences by looking across Talent Search programs. Certain 
differences across programs could be related to different placement rates. One type of 
differentiating characteristic would be the particular goals that projects may have. For example, a 
project whose main goal is to ensure students' graduation from high school might not expect to 
produce high college placement rates. 

In exploring the possibility of comparing Talent Search projects, it was considered 
important to identify clear categories of Talent Search projects along which they could be 
compared-in their goals, strategies, targeted students, or demographic characteristics. A number 
of conference participants believed this to be a worthwhile activity likely to generate valuable 
information about effective practices in Talent Search programs. 

Existing data sources. The most current Talent Search data come from the projects' 
Annual Performance Reports. The very limited interpretive power of the Performance Report 
data confirmed the general view of meeting participants that the Performance Report forms must 
be redesigned. For example, many Talent Search services listed on the form (for which grantees 
were to note whether the service was provided and the number of participants served) were so 
lacking in definition (e.g., counseling, computer-assisted instruction) that one cannot meaningfully 
describe what these activities entail and the extent to which they take place in Talent Search 



projects. Age groupings were not broken down further than age 12 to 18, so the intermediate-level 
Talent Search component could not be analyzed separately from the high school component. 
Further, the Performance Reports provide only a snapshot of a project at one point and do not 
convey the incidence of students' continuation in the project or into postsecondary education from 
year to year. 

Since cla ity of definition has been lacking in studies of Talent Search, Trent proposed 
that this evaluation's research strategy should establish clear definitions early on. He suggested 
that qualitative strategies do this better than quantitative strategies; qualitative strategies do better 
at grounding people in an understanding both of process and of operating definitions people use. 
Using qualitative strategies early on allows one to develop better definitions and indicators to use 
in the survey research stage. 

Trent also recommended using data from the national surveys High School and 
Beyond (HSB), the National Educational Longitudinal Survey (NELS), and the National 
Longitudinal Survey (NLS). For example, the responses of those HSB participants who reported 
themselves as having been in Talent Search could be analyzed for purposes of descriptive 
information. Unfortunately, Department of Education participants noted, the variable pertaining 
to participation in Talent Search and similar programs contains a great deal of measurement error, 
since, at the local level, many Talent Search programs are known to students by other names. 

Time Frame for the Next Evaluation 

Angelo envisioned the Talent Search evaluation to be a multi-year study. He 
suggested that perhaps a year should be taken to design and send out RFPs for several 
demonstration evaluations of Talent fearch programs that have different characteristics or 
settings. Based on those initial findings, another RFP would be released, with the whole study 
taking five to six years. 

Angelo suggested that an overall evaluation of this program would contain four 
components or stages: 
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1. a retrospective or descriptive stage, to include the demonstrations suggested 
above; 

2. a redesign stage, involving a conference to examine all the results to date and 
consider changing program guidelines or regulations as well as designing the 
remainder of the evaluation;' 

3. a prospective stage during which the bulk of the evaluation is conducted; and. 

4. a dissemination stage, including training and outreach components, to 
incorporate recommended practices into actual Talent Search projects. 

Discussion among conference participants suggested that the time frame of the 
evaluation would depend in part on decisions to measure shorter-term or longer-term outcomes. 
If the study involves looking at postsecondary enrollment and attendance, for example, this would 
call for a longer time frame. 
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A Note to Readers 



Throughout this paper. I make assumptions, raise issues, suggest criteria, and present options that 
invite challenge and discussion. Although this may be a somewhat unusual approach, it's my hope 
that this "monologue" will serve to stimulate spirited, constructive conversations among those 
charged with designing the Talent Search evaluation. 
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Designing a Useful and Appropriate Evaluation of Talent Search: 
What Are the Most Important Design Questions to Consider? 

Thomas A. Angelo 



1. Overview and Assumptions 

What is the most effective and appropriate way to evaluate the Talent Search 
program? This paper takes a first step toward answering that question by focusing on broad 
conceptual and practical issues that must be addressed in order to design a useful evaluation rather 
than on more specific technical issues of measurement and data analysis. Because responses to 
these overall design questions will determine, in large part, the outcomes of the Talent Search 
evaluation, they need to be raised and answered first. 

"No matter how precise your measurement or how sophisticated your analyses, you 
risk failure if your research is not well planned. You can't fix by analysis what you 
bungled by design: (Light, Singer, and Willett, 1990, vii-viii, emphasis original.) 1 

One way to reduce the risk of "bungling by design, 1 ' is to begin the design process by 
raising very basic questions, such as the following: 

■ Why is the Talent Search program being evaluated now? (What are the main 
purposes of this evaluation?) 

■ For whom is it being done? (Who are its audiences?) 

■ How will the results be used? (What difference is the evaluation likely to 
make?) 

In response to the three questions above, I have made several assumptions which 
undergird the rest of the paper. If they are not valid, then what follows will surely be of little use. 



1 This quote is taken from By Design. Planning Research in Higher Education by Rschard Light. Judith Singer, and John Willett 
(Harvard University Press, 1990). In preparing this draft. IVc drawn repeatedly on their design suggestions. 



Therefore, it's important to make those assumptions explicit now, so that they can be carefully 
scrutinized and, if need be, corrected. 



Whv is the Talent Search program being evaluated now? (What are the main 
purposes of this evaluation?) 

The related questions of timing and purpose are particularly relevant in this case, 
since the last evaluation of the Talent Search (TS) program was launched nearly 20 years ago, and 
its results published in 1976. 2 I assume that the decision to evaluate the Talent Search program 
again now is motivated by the widespread general interest in assessing the effectiveness of 
federally funded programs in education. In other words, I have no reason to believe that the 
Talent Search program is being singled out for evaluation at this time because of perceived poor 
performance, or because it has been targeted to be cut for budgetary reasons. 

Further, I assume that the Talent Search program evaluation will have five main 
purposes. First, it will seek to determine - or reliably estimate - the extent to which the Talent 
Search program as a whole achieves its broad goals, at what cost, and the range of variation in 
effectiveness among local projects. Second, the evaluation will try to determine which types of 
services, of the many that Talent Search projects offer, are generally most useful and effective. 
Third, it will attempt to discover « or validate - some general characteristics of particularly 
successful Talent Search programs in order to develop useful guidelines for selection and 
implementation. Fourth, the evaluation will aim to identify, much more specifically, what kinds of 
Talent Search services and programs work best for whom and in what circumstances, once again to 
develop guidelines for more effective design and practice. And fifth, I assume that it will shape the 
program's policies and practices on record-keeping and reporting, making future program 
evaluations less burdensome and more informative. 

If these assumptions are correct, then the overriding purpose for evaluating the 
Talent Search program now is to improve its effectiveness at encouraging and assisting 
academically able but economically disadvantaged young people to successfully complete high 



2 Durham, N.C. Research Triangle Institute. Center for Educational Research and Evaluation. A Study of the National Upward Bou 
and Talent Search Programs: Final Report. Washington: Department of HEW. Elation Division. Office of Education, Office 
Planning, Budgeting, and Evaluation. 1976. 
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school and to pursue further education. Consequently, the Taient Search program evaluation 
should be more prospective than retrospective, and more formative than summative. 

For whom is it being done? (Who are its audiences?) 

In relation to the assumed purposes listed above, I also assume that the evaluation 
will be done to meet the requirements and interests of specific "audiences." In designing the 
evaluation, determining whose questions and needs it should address is critical. The four main 
audiences I envision are ranked in order of their interest in the evaluation and its likely impact on 
them. The first audience will be staff members in the Department of Education responsible for 
the Talent Search program. The second audience will be those administering ongoing Talent 
Search projects and those interested in applying for awards to start new projects. The third will be 
external "auditors," such as the OMB or CBO, and Congressional committees concerned with 
appropriations and oversight. And the fourth audience will consist of the educational evaluation 
and assessment "community," academics and professionals who may profit from studying this 
evaluation's design, methods, and results. 

How will the Talent Search program evaluation results be used? (What difference is 
the evaluation likely to make?) 

While it's necessary to keep in mind who will use the results of the Talent Search 
evaluation in order to focus it to meet specific needs, it's not sufficient. To design well, it is also 
critical to make some assumptions about how and for what those audiences will use the evaluation 
results to benefit the clients mentioned above. 

Based on my assumptions about the evaluation's purposes, I expect that Department 
of Education staff would use the results to revise guidelines for awarding grants to new and 
continuing Talent Search projects; to provide specific, practical advice and more effective 
consultation to local project directors and staff; to comply with external requests for evaluation 
information; and to guide and improve future program evaluation efforts. 
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Those administering local Talent Search projects would use the evaluation results to 
direct and strengthen their efforts, and applicants for awards would follow these guidelines in 
designing their project proposals to increase the likelihood of being selected. 

It is also possible, of course, that the evaluation results could lead the Department 
staff to propose significant changes in the structure or funding levels of the Talent Search program. 
They might even lead to the design of new programs to encourage and assist economically 
disadvantaged students to prepare for and succeed in postsecondary education. 

External reviewers and policy and decision-makers are likely to use the results of this 
evaluation to help them evaluate Talent Search program funding requests and set budget 
priorities. As demands on available federal funds continue to increase, it is reasonable to assume 
that programs which provide the Secretary of Education and Congress with understandable, 
convincing evidence of effectiveness will be more likely to have funding continued or increased - 
or, at least, less likely to have funding cut. 

Lastly, if the Talent Search evaluation is successful, evaluation and assessment 
professionals are likely to look to the results for lessons and guidelines that can be applied in the 
evaluation of similar programs. 

2. Criteria for Effective Program Evaluation 

Just as it is important to make assumptions that undergird an evaluation design 
explicitly, it is valuable to summarize the criteria one uses to judge the effectiveness of educational 
program evaluation. The following list of twelve criteria, though not exhaustive, is a synthesis of 
my personal experience and my reading of the relevant literature. My subsequent discussion of the 
Talent Search program evaluation design will reflect thesw criteria. 

To be most effective, the Talent Search program evaluation should: 

1. evaluate whar the program actually does: not just what it is assumed or 
supposed to be doing; 

2. respect reasonable and necessary diversity and differences in local projects; 
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3. actively involve local project staff, school and college personnel, parents and 
students - and others likely to be affected by the results — in designing and 
carrying out the evaluation, and in drawing lessons from and disseminating the 
results; 

4. incorporate and build on already occurring evaluation, whenever possible, to 
avoid burdensome redundancy; 

5. evaluate process as well as performance and outcomes; 

6. use multiple and varied evaluation methods and measures; 

7. be carried out at various points over a sufficient period of time to detect and 
measure changes; 

8. be sensitive and powerful enough to detect meaningful effects; 

9. provide information for improving the quality of the overall program and local 
projects to those most affected, at key points before the final report; 

10. discover, highlight, and disseminate effective practices, as well as uncovering 
ineffective ones; 

1L be an intrinsically worthwhile educational activity; that is, to the degree 
possible, the evaluation should enhance, rather than interfer with, the ongoing 
program and project operation; and 

12. consistently conform to the highest ethical and professional standards of 
program evaluation. 



3. Important Design Questions 

In this section, I present several questions which must be answered in order to design 
the Talent Search evaluation. Later in the paper, where possible, I will suggest tentative answers 
and design recommendations in line with assumptions and criteria mentioned above. 



What policy questions should the Talent Search evaluation answer? 

.Although this issue will be addressed in depth by other authors, there are some 
questions that any evaluation of the Talent Search program would likely address. First and 
foremost, the evaluation must seek to determine the degree to which the Talent Search program 
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(and/or a selected sample of local projects) meets intended objectives. These mandated objectives 
are: 



(1) To identify qualified youths with potential for education at the postsecondary 
level and to encourage such youths to complete secondary school and to 
undertake a program of postsecondary education; 

(2) to publicize the availability of student financial assistance available to persons 
who pursue a program of postsecondary education; and 

(3) to encourage persons who have not completed programs of education at the 
secondary or postsecondary level, but who have the ability to complete such 
programs, to reenter such programs, 3 " 

On the most basic level, therefore, it will be useful to evaluate how and how effectively 
a representative sample of different local projects respond to each of the objectives listed above. 
The assumption here is that any given project will be more effective at realizing some of the Talent 
Search objectives than at realizing others. For example, a given project may be particularly 
successful at encouraging students to complete high school, but less successful in getting them to 
apply on to college. 

Overall, the Talent Search program mav also be more effective at achieving certain 
objectives than others. But before the evaluators can determine how and how well projects are 
achieving Talent Search program objectives, they will need to find out which objectives the 
individual projects are actually trying to achieve and what they are doing to achieve those 
objectives. 

I propose that the evaluation be designed to answer the following ten questions. 
While the answers to each question are likely to have implications for practice, the synthesis of 
answers to all ten will have policy implications for the Talent Search program and for local 
projects. 



3 Higher Education Act of 1965 as amended by PL 96-374. 10/3 SO and PL 99-198. 10/ 17, So. Subpart 4. Sec. 417B 
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Ten Questions that the Talent Search Evaluation Should Answer 

1. To what extent are each of the Talent Search program's overall objectives being 
addressed? 

2. Who are the clients being served by the local Talent Search projects? (How 
diverse are the populations being served?) 

3. Where are the projects located and how does location affect project objectives, 
treatments, and outcomes? 

4. What specific services are local projects offering to address each of the overall 
Talent Search program's objectives? (What specific kinds of treatments are 
being applied?) 

5. When do the various services begin and end? When do clients typically take 
greatest advantage of services offered? (What are start times, durations, and 
end times of treatments?) 

6. How much of each type of service do the projects provide to clients and for how 
long? (What are the intensities and durations of treatments applied?) 

7. How effectively do various treatments and projects succeed at realizing stated 
objectives? (How well do treatments/projects actually achieve what they are 
supposed to?) 

8. How efficiently do various treatments and projects succeed at realizing stated 
objectives? (What are the relative costs - in client time, staff time, and money 

of th'j effects achieved?) 

9. What else is occurring as a result of the Talent Search projects? (What are the 
unintended effects, both negative and positive?) 

10. So what and what next? (What do the answers to the eight questions above 
reveal about the overall effectiveness of the Talent Search program? What do 
they suggest for the future?) 



What is already known about the Talent Search program/projects? 

Answering this question is a necessary first step in the evaluation process. It is 
possible that at least some of the questions listed above have been or can be answered without 
further data gathering. At minimum, finding out what is known will require a review and summary 
of recent annual performance reports, if this has not already been done. Evaluators may wish to 
focus their review mainly on performance reports from projects that have previously received 
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continuation awards, since the decision to continue funding presupposes a certain level of success 
in achieving objectives. Alternately, it may be instructive to analyze reports from several projects 
that were denied continuation grants, as well, for the sake of comparison. 

How comparable are individual Talent Search projects? 

The answer to this question will determine whether or to what extent data from 
various local programs can be aggregated. Based on the brief project descriptions that IVe read, 
and the latitude allowed by the proposal guidelines, it appears that Talent Search services offered 
(treatments) vary widely in kind, intensity, duration, and quality. Project environments, clients, 
staff and resources also differ. For these reasons, I suspect it will be very difficult to aggregate 
across projects in a meaningful way, unless projects are categorized into comparable groups and 
their effects are then compared within groups. 

Foi purposes of comparison, local Talent Search projects could be categorized in 
several different ways: by size, by geographic location, by demographic characteristics of the client 
population, by types of services offered, or by type of institutional project site (e.g., secondary 
school, postsecondary school, private agency). For this evaluation, it may be most useful to group 
and compare projects that are similar in terms of size (numbers of staff and of clients served), 
location (urban, suburban, rural), types of services offered, and types of clients (socioeconomic 
characteristics). For example, three large Talent Search projects which offered similar services to 
young people of roughly the same socioeconomic status in three geographically diverse inner cities 
such as Oakland, St. Louis, and Newark, might be compared. 

That said, it should still be possible to compare the performance of all or most Talent 
Search projects in terms of a handful of indicators related to program objectives. For example, it 
may be possible to compare how effectively all or most programs succeed at "identifying qualified 
youths with potential for postsecondary education," or at "publicizing the availability of student 
financial aid." By analyzing performance reports, evaluators should be able to isolate a few 
common, comparable indicators. 

It is desirable to find common indicators, whenever possible, because program-wide 
comparisons have the advantage of large sample sizes, and thus, of greater statistical power to 



detect effects and greater generalizability of lessons learned. It is also valuable to identify and 
measure some useful program-wide performance indicators in order to respond to external 
stakeholders. Program-wide results based on large samples have more "face" validity. On the 
other hand, the level of generality of these program-wide comparisons, and the range of local 
diversity they may mask, will probably make them of limited use for understanding "what works for 
whom, when, and where" and, consequently, for improving the quality of local projects. 

It may be possible to answer the comparability question through document analysis 
alone. It is likely to be necessary, however, to follow-up the document analysis with in-depth 
telephone interviews - and perhaps with one- or two-day site visits - to check the reliability of the 
reports and to determine what other sources of information are available. 

What does "success" mean in terms of the Talent Search program? 

At this point, it will be critical to begin to define what "success" means in the Talent 
Search program. (Substitute "effectiveness" for "'success" if you wish.) The meaning of "success" 
will, of course, be relative to the program o%*erall, to the particular program objective being 
evaluated, and to the resources and circumstances of each local project. 

First, success in the Talent Search program needs to take into account the varied but 
relatively limited "treatments" it offers, and its low per-student cost. Outcomes are likely to be 
modest, overall, even for very "successful" projects. Second, success in identifying "qualified youths 
with potential for postsecondary education" and success in encouraging them to persist and 
graduate from high school and to go on to college should not be measured on the same yardstick. 
The latter objective is more difficult to achieve than the former, so success should be scaled 
differently for these two objectives - and probably for the others, as well. For instance, all 
"successful" projects might be expected to identify 90 percent of all eligible potential clients. But to 
expect a 90 percent success rate in terms of students served graduating from college would be 
unrealistic. And third, success in some environments and with some populations is likely to be 
much more difficult to achieve. It is harder and more expensive, after all, to build a skyscraper on 
landfill than on bedrock. Therefore, the overall socioeconomic status and academic preparation of 
the clients has to be factored into the calculations of success. 
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Any calculation of the success of a Talent Search project should take inputs and 
program environment into account, not just outputs. Therefore, the definitions of success used in 
evaluating Talent Search projects should, at least to some extent, be conditional and tied to specific 
objectives. One size won't fit all. And the various "yardsticks" or scales against which success is 
measured will probably serve only to make reliable and useful distinctions between "low,* 1 
"average," and "high" levels of success. Finer distinctions require more precise and accurate 
measurements than may be possible in this evaluation. 



Should the evaluation design include a control or comparison group? 

It is unlikely that a strict control group can be set up in the context of a complex, 
ongoing Talent Search project, but comparison groups certainly can and should be used. Where 
possible, students who are eligible for Talent Search services who elect to take advantage of them 
should be compared - in terms of the outcomes discussed below - with otherwise comparable 
peers who choose not to make use of the Talent Search services offered them. Similarly, students 
who are eligible for the program, but cannot be served simply because of enrollment limits, can be 
compared with those who made it in "under the limit" if the two groups are comparable in other 
important ways. 



4. A First-Draft Evaluation Design for Talent Search 

Although it is neither possible nor desirable, at this point, to design the Talent Search 
evaluation in detail, it may be useful to have a first-draft design proposal to critique. To advance 
the design discussion, I propose the following outline (Table 1) for a six-year, multi-site evaluation 
of Talent Search: 
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Table L Proposed outline for evaluation of Talent Search 



Phase 



Primary Activity 



Goals of Phase 



Year(s) 



Phase I Analyzing annual reports and 

other project documents 



Phase II Interviewing staff and/ or 
visiting sites of "successful" 
projects; Holding focus-group 
interviews with students, 
parents, project staff, school 
and college personnel 



Phase III Creating new, interim pro- 

posal guidelines and selection 
criteria for exemplary projects 
to include in sample for next 
phases of evaluation; Dis- 
seminating those guidelines 
and criteria through a variety 
of means 



Phase IV Designing next phases of the 
Talent Search evaluation in 
detail 



Phase V Selecting samples of new and 
continuing "exemplary" pro- 
jects to evaluate over time 



Phase VI Monitoring and evaluating 

sample of new and continuing 
"exemplary" projects 

Phase VII Analyzing, disseminating, 

and implementing evaluation 
results in annual and final 
reports, other formats, and 
through various media; 
Following up with training 
and support 



To determine what's already Year 1 

known about what works, in 
general; to de fine/refine 
questions for follow-up 
interviews and site visits 

To validate results ofdocu- Year 1 

ment analysis; to beein to 

determine what works for 

whom in which contexts; to 

summarize results of Phase 

I and II for use in revising 

guidelines 

To use lessons from Year 1 

preliminary results to 

influence and improve 

proposals for new and 

continuing projects; to 

develop selection criteria 

for a sample of more 

"evaluable" exemplary 

projects to evaluate 

To create a final evalua- Year 2 

tion design that embodies 
"effective evaluation " cri- 
teria and responds to 
audiences ' end clients ' 
needs 

To select a sample of well- Year 2 

documented "exemplary" 
projects large and diverse 
enough to answer questions 
well and with confidence 

To collect vaiid reliable, Years 3, 

detailed and useful data for 4 and 5 
3 years on sample 

To regularly 'feed-back" Years 4, 

evaluation results in forms 5^ an d 5 
that its audiences can use 
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A More Detailed Look at the Proposed First-Draft Evaluation Design 
Phase I: Document Analysis (Year 1) 

The only way that specific, realistic success scales can be constructed is by locating, 
analyzing, and describing the results of a range of different Talent Search projects. First, the 
evaluator(s) should assemble a large, representative sample of recent projects for which good 
documentation is available ~ regardless of their reported success or failure. What "large" means, 
in this context, will be determined by the amount of time and labor available for this stage of the 
evaluation, the number of projects which have high-quality documentation, and the number of 
broad categories that Talent Search projects fall into. 

Careful descriptive analysis of annual project reports should uncover the range of 
outcomes in relation to each program objective for Talent Search projects of different types, sizes, 
scopes, and settings. Once the ranges, central tendencies, and variability of outcomes are 
described, expert judgement will be required to peg the appropriate levels of success for each scale. 

Whatever the specific information that results from this type of document analysis, 
the exercise is almost certain to result in more detailed and more useful reporting guidelines. Put 
another way, it is only when you evaluate that you find out what kind of record-keeping you should 
have been doing all along in order to be able to evaluate well. 

Phase II: Follow-up Interviews and Site Visits (Year 1) 

A good second step would be exploratory telephone interviews with the relevant 
project directors and staff to find out what documentation or information they may have that is not 
included in the annual reports. If the original sample is too large, it may be necessary to select a 
representative subsample for follow-up interviews. Mail surveys may be used to supplement or 
supplant telephone interviews. Mail suiveys should not be used, however, unless ail possible steps 
have been taken to ensure a relatively high response rate. 

Once projects have been located that appear to work well - that is. projects that 
provide good evidence of achieving average or high levels of success in meeting objectives - then 
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the evaluation should determine what it is, in general, that is responsible for that success. In other 
words, when Talent Search projects work, what is it about the projects that works? Are there 
common characteristics shared by successful projects? To answer these questions, evaluators will 
likely have to organize individual interviews and focus-group sessions with representatives of all 
the various groups involved in local Talent Search projects. This will mean interviewing and/or 
meeting with groups of TS project administrators, counselors, tutors, support staff, high-school 
counselors, college admissions and registrar's staff members, teachers, parents and - last and 
clearly not least - eligible students who participate in the local projects and those who choose not 
to. 

The U.S. Department of Education s 1988 Application Development Guide states that 
"Experience has shown that most successful Student Support Services (SSS) projects have certain 
common characteristics. 1 It then goes on to list "characteristics of successful [SSS] program 
practices." Among these characteristics are a strong institutional commitment, high standards for 
participants, mechanisms for monitoring student performance, and follow-up. Appendix A lists 
these characteristics. 

The Guide does not indicate whether this summary list of common characteristics of 
successful projects is an outcome of evaluation research or the distillation of empirical 
observations, or both, or neither. At the same time, this list is presented as a summary relevant to 
SSS projects in general, not only to the Talent Search program. Nonetheless, the characteristics 
listed are quite reasonable ones and are likely to correlate with success in almost any similar 
educational progra m . 

This list of characteristics provides a useful starting point, a checklist that can be used 
in evaluating individual Talent Search projects for common characteristics that may be related to 
success. This segment of the evaluation could serve to confirm (and/or disconfirm) the relevance 
of the general list to successful Talent Search projects. As evaluators noticed the presence of some 
of these program characteristics, they could begin to describe them much more specifically in 
relation to Talent Search projects. For example, this type of analysis might answer questions 
such as: What particular kinds of institutional commitments, staffing, standards, monitoring 
mechanisms, and follow-up seem to work for Talent Search? At the same time, evaluators would 
need to look carefully for more specific characteristics common to successful Talent Search 
projects which were not among the characteristics listed. 



53 



40 



It may also be useful to analyze a sample of projects that were clearly not successful. 
This qualitative analysis of the characteristics of particularly successful (and unsuccessful) Talent 
Search projects should lead to a more refined and relevant project evaluation checklist, and to 
more useful guidelines for those proposing, selecting, and directing these projects. 

As a third step in this phase, the evaluation should try to tease out how well the 
program/project worked for whom, and under what circumstances. For example, is the Talent 
Search program more effective for women (who make up the majority of clients) than for men, in 
general, or is it more effective for native-born U.S. students than for immigrants? Is it more 
effective in certain regions than in others? Is Talent Search more successful in getting students to 
enroll in college if they first encounter the program in middle school rather than in high school? 
Does Talent Search work better on campuses where there are also other TRIO programs? The 
specific variables the evaluation should focus on can best be determined during Phases I and II. 

What outcomes should the Talent Search evaluation focus on? 

Given the project objectives quoted above, the document evaluation, follow-up 
interviews and site visits should, at minimum, focus on outcomes such as the numbers, percentages, 
and demographic profiles/types of: 

■ potential Talent Search student participants identified as eligible to participate; 

■ students identified as potential clients who were or were not served by Talent 
Search; 

■ clients served who did or did not successfully complete high school; 

■ clients served who did not complete high school who did or did not 
subsequently earn the GED; 

■ clients who did/did not successfully complete college-preparation courses and 
curricula (and how many courses and which courses); 

■ clients who successfully completed college preparation courses who did or did 
not enroll in postsecondary programs; 

■ types of postsecondary institutions and programs enrolled in by those who went 
on; 
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u clients who enrolled in postsecondary programs who did/did not receive 
financial aid; 

s clients who enrolled in postsecondary programs who successfully completed 
their programs of study; 

■ types of programs clients completed; and 

■ destinations of those who did not complete their programs of study. 

Phase III: Creating New Proposal Guidelines and Sample Selection Criteria (Year 1) 

Once the initial analysis of data collected in Phase I and II is complete, the 
evaluator(s) should base new, interim Talent Search project proposal guidelines on lessons 
learned. It might be wise and useful to involve groups of Department of Education regional staff, 
project directors, counselors, high-school and college personnel, parents and students in this 
process. These "stakeholders" could be involved in suggesting guidelines and/or in critiquing 
guidelines proposed by evaluators and Department of Education staff. 

These new guidelines would encourage proposals for several different types of new 
and/or continuation projects in order to create a diverse sample of evaluable, comparable 
exemplary projects. Thus, these guidelines would make use of initial evaluation results both to 
improve the overall quality of project proposals, and to elicit a sample of proposals for projects 
which could be carefully evaluated over a three-year period. 



Phase IV: Designing Next Phases of the Talent Search Evaluation (Year 2) 

At this point, after an entire year of retrospective evaluation and analysis of results, 
and after the development of selection criteria for new and continuation projects to form the 
evaluation sample, it will be possible to design the final three phases and next four years of the 
evaluation. During this phase, the evaluation designer(s) will make final decisions about sample 
size and composition, indicators and measures, methods of data collection and analysis, and means 
of disseminating i esults. 
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Phase V: Selecting the Sample of Projects to be Evaluated (Year 2) 

After the evaluation design has been completed, the evaluator(s) will select the 
sample of local projects to be monitored and evaluated over the following three years. 

Who/what should be included in the sample? How large should the sample be? 

If the evaluation is to identify and highlight good practices and improve the 
effectiveness of the Talent Search program overall, then it should focus, at least in large part, on 
successful or "exemplary" projects. By promulgating new proposal guidelines, the Talent Search 
program can, in effect, create a sample of new and continuing projects to evaluate. Initially, 
successful continuation projects will have to be selected primarily on the basis of information 
provided in their own annual reports and continuation proposals. This means that quality of 
record-keeping and reporting may be confused or conflated with program quality. The degree to 
which these projects actually were successful will have to be determined carefully during the early 
phases of the evaluation. 

As one example, it may prove useful to evaluate the relative effectiveness of "early 
intervention" TS projects, especially since recent increases in program funding have been 
dedicated, in large part, to new projects serving middle-school students. Perhaps the sample of TS 
projects can be clustered into "early-", "mid-", and "late-intervention" designs, depending on 
whether services begin in grades 7 or 8, 9 or 10, or 11 or 12, respectively. Design types might also 
be clustered to represent differences in the intensity and/or duration of services offered. 

New projects will need to be selected according to the degree to which their proposals 
reflect the revised guidelines. That is, their designs should clearly reflect lessons learned from the 
first phases of the program evaluation; they should be designed for easy monitoring and 
evaluation; and they should embody particular promising "design types," the effectiveness of which 
the Talent Search program has decided to evaluate. 

If relative success and design are the first selection criteria, then project size should be 
the second. Only relatively large projects will involve enough students to allow for meaningful 
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evaluation over time. Having relatively large sample sizes will be particularly important in teasing 
out whether treatments and outcomes do differ in important ways from site to site. 

After success, design, and size, the demographic characteristics of the client 
populations should be considered in selecting projects. Project sites should be selected which 
represent the racial, ethnic, and socioeconomic diversity of the overall Talent Search program 
client base. The demographic characteristics of local projects may be closely linked in many cases 
to their sites. In any case, both rural and urban sites should be included in the sample. 

Phase VI: Monitoring and Evaluating the Sample (Year 3, 4, and 5) 

This will be the most labor- and time-intensive, and most expensive phase of the 
evaluation. For three years, evaluation teams will need to continuously monitor, analyze, and 
report on the sample projects. This will require regular document analysis, live and telephone 
interviews, site visits, and follow-up mail surveys - as well as a continuous flow of information 
through electronic and postal channels between project sites and the evaluation center. If possible, 
a handful of projects - each representing a different "cluster" of the sample -- should be studied in 
depth to serve as subjects for case studies. Rich narrative data from these in-depth qualitative 
studies will help to supplement and explain the quantitative data. 

Phase VII: Analyzing, Disseminating, and Implementing (Years 4, 5, and 6) 

The final phase will, in many ways, be the most important one. At this point, the 
evaluator(s) must summarize what they have learned from several years of evaluation and explain 
its implications for the various audiences the evaluation was designed to serve. The final products 
of the Talent Search evaluation should be a set of proposed guidelines for revamping the proposal 
selection, implementation, monitoring and evaluation of all future Talent Search projects. 

These "lessons learned" should be adapted to and shared with the full range of TS 
clients and audiences at all levels. It may be useful to involve teams of project directors, 
counselors, high-school and college personnel, parents and students in designing and producing 
many types of "final reports" to communicate with and disseminate findings to these various 
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groups. Results should be disseminated through various media - not just print - and might 
include the production of video-tapes, interactive videodisks, and computer software. 

If local projects are to apply lessons learned in practice and not merely read or hear 
about them - the ED and Talent Search program staff will need to sponsor regional conferences, 
develop and lead training workshops, and engage in follow-up surveys, interviews, and site visits. 
Ongoing assessment and communication must be built-in to new TS projects, and across the 
program as well. 

In the final analysis, the success and value of the Talent Search program evaluation 
will depend on the effectiveness of dissemination and follow-up efforts. 
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SUGGESTIONS FOR FISCAL YEAR 1990 APPLICANTS 
FOR STUDENT SUPPORT SERVICES FUNDS 



Experience has shown that the most successful Student Support Services projects have certain 
common characteristics. These characteristics may be summarized as follows: 

1 Projects which have a strong institutionalized commitment to their objectives. This 
often takes the form of in-kind or cash contributions to enhance the opportunities 
which are available to students through the Student Support Services project. 

2. Projects which are fully understood by and which work closely with all of the 
administrative and academic departments of a participating inst'tution. 

3 Projects which provide mechanisms for continually monitoring student performance, 
both in project sponsored academic programs and in regular course work being 
undertaken at the institution. 

4 Projects which establish high standards and expectations for students including the 
belief that all students regardless of family background, can reach high levels ot 
academic achievement. 

5 Projects which follow up on their Student Support Services "graduates" by monitoring 
the progress and performance of those who have entered another postsecondary 
educational institution or graduate school. 

6 Projects which aive prioritv to the strengthening of basic and higher level skills of 
their Student Support Services participants in mathematics, science. English language 
literacy in reading, writing, and speaking, and foreign language literacy. 

7 Projects that actively seek to improve equal educational opportunity and access for all 
students particularly those who traditionally have not participated fully m higher 
education, including'projects that address the special skill needs of members of racial 
or ethnic minority groups, women, and the handicapped. 

8. Projects which specify a method of documenting eligibility, selection, participant need, 
services provided and participant success. 



CRITERION: THE SECRETARY LOOKS FOR INFORMATION THAT SHOWS THAT THE 
BUDGET FOR THE PROJECT IS ADEQUATE TO SUPPORT THE PROJECT 
ACTIVITIES. (34 CFR 646.31(c) (2) (i)) 



"Adequacy" is a judgment that the field readers will make based upon the information 
you provide in the program narrative and the budget. You may find it helpful to 
include the institutional salary scale. 



CRITERION: THE SECRETARY LOOKS FOR INFORMATION THAT SHOWS COSTS 
ARE REASONABLE IN RELATION TO THE OBJECTIVES OF THE 
PROJECT. (34 CFR 646.31 (c) (2) (ii) 



"Reasonableness" is a judgment that the field readers will make based upon the 
information provided in the application. 

Address and justify the reasonableness of the request. 
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1. Introduction 

Disadvantaged students who earn college or other postsecondary degrees follow a 
different trajectory from that of their parents before them and are in a better position to support 
and serve as role models for their own children. For this reason, the federal government has taken 
an active role in assisting disadvantaged youth to achieve success at the postsecondary educational 
level. 

This assistance has been directed, primarily, toward removing financial barriers to 
college education. Among these programs are the GI Bill, the Perkins Loan Program (formerly 
the National Defense Student Loan Program), Guaranteed Student Loans, College Work-Study 
programs, and Supplementary Educational Opportunity Grant programs. 

While authorizing these programs. Congress has also recognized that disadvantaged 
students face non-financial problems that decrease the likelihood that they will even consider 
college. It is now clear that the decision to pursue higher education is a complicated one, involving 
prior preparation reaching back years before high school graduation. 

Academic preparation for college requires selection of particular classes as early as 
the ninth grade, and school completion requires persistence in the face of many obstacles, 
including peer pressure against academic effort. Not only can the cost of college tuition and room 
and board appear daunting, so too can the financial aid system - particularly to students from 
economically disadvantaged backgrounds who are the first in the family to attend college. 

It is therefore important that students and their parents are provided with early and 
realistic information about the costs of higher education, and the ways of meeting them, so that the 
students are free to aspire to college and concentrate on the necessary preparation. 

This is rarely the case for students from disadvantaged families, however. Instead, 
these students and their parents have limited knowledge of the costs of attending different kinds of 
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schools - both grossly overestimating and underestimating different cost elements. They also 
know surprisingly little about the availability of federal student aid, even into their junior and 
senior years. High school counselors are not generally regarded as important sources of financial 
aid information. 

Talent Search Program 

In order to address some of these non-financial barriers, Congress passed legislation 
to establish programs to recruit disadvantaged students, prepare them for postsecondary 
education, and assist them in seeking financial aid. The Higher Education Act of 1965 established 
the Talent Search and Upward Bound programs and three years later, the Special Services 
program (now known as the Student Support Services program) - and designated these efforts the 
Trio Programs. While Talent Search and Upward Bound helped students gain admission to 
college, Special Services was aimed at helping disadvantaged students adjust to college life and 
complete a collegiate program. 

Talent Search was designed to identify qualified youth with the potential for 
postsecondary education, to encourage them to complete secondary school and to enroll in 
postsecondary education programs, to publicize the availability of student financial aid, and to 
increase the number of secondary and postsecondary school dropouts who re-enter an educational 
program. It serves low-income individuals (from families with incomes less than 150 percent of 
poverty) between the ages of 11 and 27 who are also potential first-generation college students. In 
a given program, two-thirds of all participants must meet a joint low-income and first-generation 
college criterion. 

In FY 1991, there were 295 Talent Search projects operated by institutions of higher 
education - community colleges and state universities primarily - and community-based 
organizations. They served in excess of 285,000 individuals with some combination of the following 
services: 

■ assistance in completing college applications, financial aid applications, and 
preparation for admissions tests; 

■ information on student assistance; 
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■ information on postsecondary education; 

■ academic, financial, or personal counseling; 

■ career exploration and aptitude assessment; 

■ exposure to a range of career options; and 

■ assistance with the re-entry process to high school or college. 

The range of services is broad, and the average per capita cost of $209 suggests a low intensity of 
service delivery. 

In the two or so decades of its history, there has been a slow, steady increase in 
funding for Talent Search. It began at $2.5 million in 1967 and reached a plateau of S27 million in 
1990. In 1991, however, there was a significant increase to S59.5 million - to support a 1989 
mandate to make assisting seventh- and eighth-grade students with such services as mentoring, 
enhanced parental involvement, and study skills, and with high school follow-up as a priority. 

Many questions remain unanswered: What services and activities do these programs 
actually deliver? Do these programs, as designed and delivered, meet the needs that exist? Have 
these programs helped advance the goal of equal educational opportunity? How could they be 
more effective? What improvements could be made in the legislation and regulations governing 
these programs? 

2. Lessons from Existing Research 

To date, there have been few comprehensive evaluations of the Talent Search 
program. The first, and most comprehensive, was completed in 1975 by the Research Triangle 
Institute. Designed to provide descriptive data on the scope and nature of the program and its 
operations, the educational background and training of key project staff, the general characteristics 
of its clientele, and the services provided to that clientele, it did not attempt an evaluation of the 
impact of the program on its participants. 

The study concluded that (among other things): the differences among Talent Search 
programs exceeded their similarities, making it difficult tc identify a typical Talent Search 
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program. Nevertheless there were a number of strengths that could be identified, including the 
apparent recruitment of a large number of eligible target youth; a core set of treatments, including 
dissemination of information, assistance in the application process, obtaining financial aid, and 
personal counseling; the effective relationships developed with a standard group of institutions to 
which targeted youth apply and many then attend; good relationships with referral agencies; 
dedicated staff; and positive influences on high school counseling programs and a variety of 
postsecondary institutions. 

Among the identified weaknesses were the fact that non-eligibies were also served; a 
mismatch between the clients and the institutions that they attend; follow-up of clients once they 
leave the program; the level of cooperation with high schools, particularly high schools that had 
come to depend on Talent Search to take over counseling of disadvantaged students; lack of ability 
to deal with academic counseling, guidance, and testing; funding; recordkeeping; and the lack of 
national visibility. 

The ability to reach such conclusions was exacerbated by: 

■ an imprecise definition of the target population and a lack of measurable 
objectives, such that virtually anyone requesting assistance could be served; and 

■ the limited utility of the data submitted to the Department of Education. The 
problem ranged from a lack of concern for accuracy, to faulty recordkeeping, to 
varying interpretations of the meaning of a client or a contact. Thus, it was not 
possible to identify the actual number of clients served and the exact nature of 
the services provided, or the number of dropouts prevented and/or returned to 
the educational system. 

The authors believed that the low probability that valid data could be obtained, and 
the variability in the implementation of the program made Talent Search a program that, from an 
impact analysis perspective, was "unevaluable." 

A decade later, a second evaluation of Talent Search reached many of the same 
conclusions. A less comprehensive study, it drew from a sample of only 17 Talent Search projects 
and reported on an in-depth analysis of the experiences of 11. It nevertheless concluded that since 
t*ie RTI study. Talent Search projects had increasingly come to deliver their services in a strategic 
cost-effective, and professional manner. It identified two prevailing service strategies: a long-term 
developmental strategy' that focused on early intervention and providing encouragement. 
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counseling and other support for students before they begin to select inappropriate high school 
courses or downgrade their educational aspirations, and the dominant strategy - a short-term 
focus on high school seniors or juniors who were already college-bound. Yet, inadequate data of 
the sort discussed in the RTI evaluation were said to have precluded a more definitive assessment. 



3, Issues for a New Evaluation 

There have been no subsequent evaluations of Talent Search. The need for one, 
however, is clear. Talent Search continues to draw upon public funds (recently at an accelerating 
rate), even though its effectiveness has never been established. Moreover, recent changes in the 
policy environment make the acquisition of timely and credible evidence about program 
effectiveness even more important. Once one of few programs seeking to increase the college 
access of disadvantaged youth, Talent Search and the other TRIO programs have now been joined 
by a group of newly emerging programs that embody competing views about how best to Improve 
college access and retention for disadvantaged students. 

One example is the "I Have a Dream" program pioneered by Eugene Lang. Following 
his lead, the private sector has become involved in providing college opportunities to 
disadvantaged children - sponsoring classes of sixth or seventh graders by guaranteeing payment 
of college tuition in exchange for high school graduation. In addition, these sponsors provide 
support services such as tutoring, counseling, and mentoring to help these youth remain in school. 

It is issues such as those identified above - the lack of firm evidence regarding 
program effectiveness, and the challenge posed by alternative programs with philosophies that 
emphasize early intervention, tuition guarantees, and continuity of care - to which an evaluation 
of Talent Search should be responsive. Below are the questions that, in light of the above 
discussions, should be asked of the Talent Search program. 



3.1 Implementation Analysis Issues 

An evaluation of Talent Search should start with a documentation of current 
operating practices. Such a study would meet the basic knowledge development function of 
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updating information about its participants and practices. Since its last comprehensive evaluation, 
there have been significant changes in the program itself, as well as in the program and policy 
context in which it operates. Among the changes in internal operations are some fairly dramatic 
shifts in the target population served. 

The Research Triangle evaluation characterized Talent Search's target population as 
disadvantaged youth who demonstrated "exceptional potential to do college-level work" (emphasis 
added). It is unlikely, however, that the majority of the youth currently being served by this 
program meet this stringent academic standard. While program operators are required to certify 
the economic eligibility of participants, they are not required to demonstrate that they have limited 
participation to students with strong potential to do college-level work. As a result, little is known 
about the academic characteristics of Talent Search participants. Preliminary evidence -- an 
examination of a number of Talent Search proposals, and a conversation with a program operator 
broadly knowledgeable about the youth being served by Talent Search - suggests that Talent 
Search does not serve the "stars" avidly sought by guidance counselors and college recruiters. 
Instead, they appear to serve the youth who might slip through the cracks « though exhibiting a 
lower level of academic performance, they nevertheless aspire to careers that require 
postsecondary education and could benefit from support and guidance. 

This shift is important. Youth whose academic performance is average or below 
average will certainly require different kinds of services, or a different level of intensity of service, 
from those intended for students with exceptional ability to do college work. The exceptional 
student may need financial assistance and college information; an average or below average 
student may also need intensive academic remediation. 

Another shift in its target population came via legislative mandate, in 1989, when 
Talent Search programs were first asked to make services to seventh- and eighth-graders a 
priority. As a consequence, it went from a program that dealt primarily with older students and 
the secondary and post-secondary educational programs in which they were found to one that had 
to deal, additionally, with early adolescents and the middle schools in which they are found. 
However, little is known about the practices that characterize Talent Search's service to this target 
group — or the appropriateness of these practices. 
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A study of program implementation should provide a complete description of the 
academic and other characteristics of the youth served. It should also delineate the program's 
services that are available to them through Talent Search, and make a determination of the 
appropriateness of these services for target youth - both the original and the new target groups. 
The questions to be addressed and the information to be gathered include the following: 

■ Whom does Talent Search serve? What are their academic, economic and 
other characteristics and, by extension, what do they need if they are to reach 
their educational goals? 

■ Does it serve youth who would not otherwise be served, or who are not being 
adequately served by guidance counselors or by other pre-collegiate programs? 

■ At ground level, what actually occurs in a Talent Search program? What are its 
services, and how are they delivered? How do they differ from services 
available in other precollegiate programs? At what level of intensity are these 
services delivered? 

■ Are Talent Search services appropriate given the needs of its participants? Is 
Talent Search uniquely able to meet the needs of this population? 

■ What other services do Talent Search participants take advantage of? 

For the benefits that can be attributed confidently to participation in Talent Search, 
what are the costs? Costs for Talent Search run about $209 per capita, approximately 1/14 of the 
costs of Upward Bound. What is being achieved for these funds, and dollar-for-dollar, is Talent 
Search as efficient as other programs seeking the same aims for the same populations? 

A recent survey of Talent Search programs noted the wide variation that existed in 
the content, quality, intensity, and duration of services offered by the responding programs. While 
recognizing the value of local variation, the authors nevertheless expressed an interest in distilling 
from that variety a set of standards and practices that could undergird all Talent Search programs. 

An implementation study that is responsive to the questions outlined above - noting 
both typical and exemplary program practices - would be a first step toward this goal. It would 
allow decision-makers to identify effective program practices that should be adopted more 
generally throughout the Talent Search network, as well as ineffectual practices that should be 
eradicated. 
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Community and Institutional Context Issues 



Beyond an analysis of the implementation of Talent Search itself, it might prove very 
useful to undertake an examination of the institutional and community context in which it 
operates. Such an examination could shed light on a number of issues important to an 
understanding of the value of this program, among them: 



Talent Search's relationship to the middle schools, high schools, community- 
based organizations, and institutions of higher education in the communities 
in which it operates. Is there collaboration and coordination between Talent 
Search and these organizations? Does coordination of this sort lead to 
improved service delivery? 

The relationship among Talent Search programs, when several operate in the 
same area. 



■ The percent of eligible youth served by Talent Search. 

■ Between Talent Search, and the various other pre-collegiate programs 
operating within communities, what percent of the Talent Search-eligible 
population receives pre-collegiate services? Do virtually all eligible youth 
receive such services, or do some slip through the cracks? 

■ Are there communities that are not served by Talent Search, or any pre- 
collegiate program, where new Talent Search programs could be sited? Are 

there communities where, in spite of the existence of Talent Search an s other 
pre-collegiate programs, a large percentage of the eligible population is not 
being reached? 

■ The relationship between Talent Search and Upward Bound and other TRIO 
programs. These programs are often sited in the same cities, and frequently 
within the same institutions. In addition, Talent Search and Upward Bound 
have similar eligibility requirements: while Talent Search can serve these youth 
at an earlier age, both serve high school students at or above the 10th grade. At 
these grade levels, are there differences in the students recruited by each of 
these programs? Do these programs differ in the types of services provided, or 
in the intensity with which they are provided? Do they produce similar 
outcomes? If the outcomes differ, which program is responsible for facilitating 
the stronger outcomes, and why? Do some of the same youth enroll in both 
these programs at different stages of their educational careers and, if not, could 
they benefit from doing so? 

The insights issuing from this inquiry could suggest ways in which more students could be served, 
service delivery improved, or costs reduced 
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3.3 Impact Analysis Issues 

Ail evaluation of Talent Search should also determine whether Talent Search has a 
significant impact on its participants, the conclusions of the Research Triangle study 
notwithstanding. If, for instance, the quality of site-level data remains an obstacle to an impact 
analysis, the development and implementation of a uniform set of descriptors and outcome 
measures should be included in the overall evaluation design. 

To date, there is no evidence to suggest that participation in Talent Search 
significantly increases a youth's chances of graduating from high school and pursuing 
postsecondary education over and above what they would have been in the absence of that 
participation. 

Earlier studies suggest that this might be the case, but they are generally based on 
inadequate design. There cannot be an adequate evaluation of the Talent Search program 
without: 

■ comparison data that say what would have happened in the absence of the 
program; 

■ information about the college readiness - including academic preparation and 
motivation to pursue higher education of both the Talent Search participants 
and that of the students to whom they are compared; and 

■ follow-up data that indicate that Talent Search participants - once admitted to 
postsecondary education - persist beyond initial enrollment and receive a 
college degree or, in the absence of graduation, gain access to better jobs. 

Without this information, one should not be impressed that any given percentage of 
Talent Search participants, no matter how high, subsequently enrol! in college. 

While an impact analysis clearly seems indicated careful consideration should be 
given to the exact nature of the impact analysis that is undertaken. To craft an impact analysis 
strategy that will provide policy makers with a picture of the strengths and weaknesses of this 
program, the following decisions should be considered: 
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The outcomes to be investigated. In theory, a wide range of outcome measures seem 
appropriate since Talent Search's reach encompasses virtually all educational outcomes associated 
with a youth's academic career - starting with middle school and ending with college graduation. 
The full range of these outcomes will apply to the sixth through eighth graders who participate in 
the program on a more or less continuous basis. 

For instance, Talent Search may influence middle school students' attendance, grades, 
and choice of high school (comprehensive versus magnet school or exam school) Later on, it may 
influence not only educational aspirations, academic performance (grades and scores on 
standardized tests), and school retention or attrition, but also whether a student pursues an 
academic track as opposed to a general or vocational track; takes gatekeeper courses in math, 
science, and languages; takes advantage of opportunities for remedial or advanced instruction; 
proceeds through high school in four years; and enters pestsecondary school immediately after 
leaving high school. 

Later, participation in Talent Search could influence choice among and acceptance in 
four-year college versus community college versus postsecondary vocational or proprietary schools. 
Among colleges, the program could influence choice between and acceptance in private colleges 
and universities versus public colleges and universities, and the amount of scholarship dollars 
offered. 

The traditional way of thinking about these outcomes is to consider admission to 
college a mere valuable outcome than admission to a postsecondary vocational school; admission 
to a four-year college better than admission to a community college; and admission to a private 
college or university better than admission to a public college or university. Another way of 
thinking about these outcomes would be to ask: What kind of postsecondary educational 
experience is most appropriate for which of Talent Search's subgroups? That is. what is the most 
rigorous educational experience to which a given Talent Search participant can gain admission and 
have a high probability of attaining a degree? For instance, is there a subset of Talent Search 
participants for whom graduation from a community college should be considered the most 
appropriate outcome? 

Beyond enrollment in postsecondary education, the evaluation should also investigate 
the impact of Talent Search on retention in college, and degree attainment. Research - on 



Upward Bound (Burkheimer, 1979; Myers, 1991) and, more recently, the High/Scope Institute for 
IDEAS (Oden, et. al. 1992) - suggests that postsecondary education programs for at-risk youth 
may improve college access, but have no impact on their performa ce in college or the likelihood 
that they will remain enrolled. To what degree is this likely to be true for Talent Search 
participants? It is also important to understand what happens to Talent Search participants who 
drop out of college without acquiring a degree. Are their employment outcomes enhanced because 
of their increased educational attainment? 

For students who enter Talent Search later - as juniors and seniors, for instance - 
only those outcomes related to college admission, enrollment, and retention are really applicable 
since many of the earlier (and, perhaps, most critical) decisions will already have been taken 
before they enroll in the program. 

Thus, conducting an impact analysis that stratifies the sample into three groups - 
those beginning participation in the 6th through 8th grades (early intervention); those beginning 
participation in the 9th and 10th grades (intermediate intervention); and those beginning 
participation as juniors and seniors (late intervention) - might make it possible to tailor the 
impact analysis toward the most likely outcomes. It may also make sense to look separately at 
programs hosted by four-year colleges, and community-based organizations; similarly, the sample 
of programs might be stratified on the basis of urban or rural location. 

Participation Issues. Another issue that deserves consideration is the nature of 
participation, and its likely effect on program impact. The kinds of impacts that are likely depend, 
in part, on the nature, content and intensity of the services a youth receives - how early he or she 
begins participation, over how long a period participation continues, with what intensity, and with 
what degree of appropriateness for the kinds of outcomes being investigated. There is a growing 
body of data and analysis that suggest that, in the absence of an intensive, appropriate, and 
sustained intervention, one should not expect to produce long-lasting impacts in the lives of 
disadvantaged youth (Walker and Vilella-Velez. 1992). 

In order to construct an impact analysis that is a fair test of Talent Search's ability to 
meet its program objectives, one must ask whether it constitutes a treatment that is sufficiently 
robust to produce impacts on high school graduation, college enrollment and persistence, and if 
not, what outcomes are consistent with its level of service provision. A study of program 
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implementation might produce findings similar to those of the Franklin (1985) study - i.e,, that 
many programs deliver limited or one-time-only services, and many fail to deliver the academic 
instruction that will increase the youth's ability to pass entrance tests and do college-level work in a 
sustained and consistent manner. Under such circumstances, there is little reason to expect that 
long-term impacts will be achieved or, if achieved, sustained. 

This level of service provision might, on the other hand, support the development of 
short-term impacts that, if followed by continuing provision of appropriate services, could 
ultimately result in longer-term outcomes of college persistence and graduation. It might also be 
sufficient to increase the college enrollment rates of high school juniors and seniors who - by 
virtue of having reached these grade levels, and by virtue of their participation in Talent Search - 
have higher-than-usual educational aspirations and are at little residual risk of dropping out. An 
entirely different level of service provision might be required to achieve the same kinds of 
outcomes for seventh and eighth graders, who face an additional five or six years of secondary 
education before college enrollment is at issue. 

Comparison Group Strategies. To whom should Talent Search participants be 
compared, and how should this comparison group be selected? 

The most critical task in the design of an impact analysis is the selection of a group of 
youth whose outcomes can serve as a proxy for what would have happened to participants in the 
absence of the program. The random assignment of eligible applicants to treatment and control 
group conditions is generally considered the most appropriate means of obtaining comparison 
data. On the other hand, one can anticipate strenuous objections to this procedure - many of 
them rooted in the perception that Talent Search is the only source of pre-collegiate services for 
disadvantaged youth in many of the localities in which it operates, and that the denial of such 
services for purposes of research would be unacceptable. 

Should random assignment prove unworkable, there are other comparison group 
options that could be considered, among them comparing Talent Search participants to 
participants in other pre-collegiate programs, or obtaining time series data from participating high 
schools before and after the introduction of Talent Search. Before any such decision is made, 
however, some preliminary work should be completed. Currently, little is known about how 
students are recruited to Talent Search. A "pipeline" study - a study that would provide critical 



64 



information about how participants are recruited to the program, how they flow through it, and are 
sorted into those who receive minimal services versus those who receive more intensive services - 
may be indicated. This is critical information, particularly in situations where Talent Search is the 
only source of pre-collegiate services. If fewer students can be served than apply -- as is suggested 
in program reports that say the programs serve more youth than is justified by their funding levels 
- a random assignment strategy may be justified. If random assignment is not indicated, a pipeline 
study may nonetheless point to other organizations and settings where comparison group youth 
may be located. 

The comparison group strategy that is selected must also be able to take into account 
academic ability and motivation. What studies there are have compared Talent Search 
participants -- whose academic motivation and performance is strong enough to have led them to 
join the program to groups of non-participants who do not have comparable motivation or 
ability. This is a problem particularly in settings where Talent Search is said to be the only game 
in town - the more able or more motivated students find their way to Talent Search; those who do 
not are often less able, less motivated, and by extension, less likely to enroll in or persist in college, 
and thus not a suitable comparison group. 

Random assignment is the design that solves this problem best. However, a 
comparison group design "that compares outcomes for Talent Search participants with those of 
participants in other precollegiate programs should be considered. Of course, this strategy would 
only be possible in areas where there are a number of precollegiate programs in operation. 
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Perspectives on an Evaluation of Talent Search: 
Interviews with Talent Search Staff 

Ann Coles 
Executive Summary 

Interviews with Talent Search project directors across the United States produced 
general consensus on the outcomes that a national evaluation should measure. Two important 
questions emerged: 

■ Does Talent Search result in increased high school graduation rates for 
participating students as compared to similar students who have not been 
involved in this or other programs to facilitate access to postsecondary 
education? 

* 

■ Does Talent Search increase the postsecondary enrollment of participants as 
compared to students who have not received Talent Search or similar services? 

Project directors interviewed also thought that the evaluation should determine the 
extent to which Talent Search participation results in financial aid awards sufficient enough to 
meet students' college costs, as compared to awards received by similar students not involved in 
the program. There also was consensus that a national evaluation should assess the degree to 
which Talent Search increases students' knowledge of career, postsecondary, and financial aid 
opportunities. 

The extent to which Talent Search heightens students* self-esteem and increases their 
motivation to pursue further education was considered important. At the same time, concerns 
were expressed about how heightened self-esteem could be measured and the extent to which 
Talent Search could be considered the primary contributing factor to increased self-esteem. 

One issue emerged on which there was considerable disagreement: whether a 
national evaluation should consider the college retention and graduation rates of Talent Search 
"graduates" as indicators of success. Some project directors felt strongly that college retention 
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should be considered. They maintained that if Talent Search indeed helps students make good 
educational choices and increases their motivation, students are more likely to succeed in 
postsecondary programs. Other project directors felt strongly that retention should not be 
considered in determining success. From their perspective, the goal of Talent Search is to "bring 
students to the college gates. 1 * Once students begin college,- however, projects have little control 
over how successful they are. Part of the reason people felt that Talent Search should not be held 
responsible for students' college success relates to the fact that projects do not have the resources 
to prepare participants for college that more intense early intervention programs such as Upward 
Bound do. 

People described critical program components in different ways. Some people talked 
about how Talent Search affects changes in students* behavior, including increasing motivation and 
awareness, building confidence, and developing decision making skills. Others defined critical 
components in terms of specific project activities such as workshops, application assistance, and 
individual counseling. 

What people would like to learn from Talent Search focused on several levels. Most 
importantly, they are interested in the broad question of whether Talent Search improves high 
school graduation rates and postsecondary enrollment rates. On another level people expressed 
interest in nuts and bolts issues related to project effectiveness. They would like to know what 
works best in terms of involving parents, improving students' academic performance, and ensuring 
that students make sound educational choices. They are also interested in the key factors 
contributing to project success, and to what extent financial resources, experienced staff, and 
program leadership are important factors in this regard. 



L Introduction 

This paper examines the views of representative Talent Search project directors 
across the U.S. on various aspects of evaluating effectiveness of their programs. It looks at what 
project directors consider to be the critical components affecting the students they serve and what 
they consider the impact of Talent Search on students, their families, the schools they attend, and 
their communities. The paper also examines the approaches that project directors think would be 
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useful in documenting program success and what they would be most interested in learning from a 
national evaluation of Talent Search. 



In developing this paper, the author interviewed the directors of 19 Talent Search 
projects. The projects from which directors were interviewed were identified in consultation with 
people knowledgeable of TRIO programs in general and Talent Search specifically. While they are 
not a random sample, the projects were selected to represent a cross-section of Talent Search 
projects in terms of geographic location, sponsorship, and environment. All the projects selected 
have been in existence for at least four years. Ten projects serve urban students, six serve rural 
populations, and three serve a mix of both. Nine projects are sponsored by community-based 
organizations, while 10 have higher education institutions as sponsors. Seven projects have 
Educational Opportunity Centers affiliated with them, while the others do not. Finally, the 
students served by the projects selected represent the diversiry of racial and ethnic groups of 
students targeted by Talent Search nationally. (In 1990-91. 33 percent of Talent Search 
participants were African- American, 26 percent Hispanic, 31 percent Caucasian. 5 percent Asian, 
and 4 percent American Indian.) 

In selecting project directors to be interviewed, people were included who have a long 
history of professional involvement with TRIO. These individuals could provide insight regarding 
similarities and differences in project outcomes over time. Some people also had extensive 
experience as trainers of Talent Search professionals, and so were knowledgeable of programs 
other than their own. 

Most interviews were conducted by telephone, although four were conducted on site 
with project directors and the staff. Interviews lasted 45 to 90 minutes each. Prior to the 
interview, each project director received a letter describing the purpose of the study and outlining 
the issues which would be discussed. Assurance was given that projects and people interviewed 
would not be identified in this paper or to the U.S. Department of Education staff. By doing so, it 
was hoped that project directors would feel free to express their views more openly than otherwise 
might be the case. 

Without exception, people willingly shared their experiences and perspectives on 
program evaluation. For many, this was one of the few opportunities the> had ever had to 
consider project evaluation in a larger content. The inter. iex\:> reflected s high degree of 
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understanding of and sensitivity to the students served by Talent Search. People interviewed also 
were interested in knowing how program effectiveness might be improved, thereby increasing the 
likelihood that project participation would result in students enrolling and succeeding in 
postsecondary education. 



2. Program Models: Similarities and Differences 

One interesting question for a national evaluation of Talent Search would be to 
determine whether different service delivery models result in different project outcomes. Because 
of the federal legislation authorizing Talent Search, the project goals of the 295 currently funded 
projects are essentially the same. The federal regulations under which the Talent Search operates 
state that the program's goal is: 

"to assist participants to continue in and graduate from secondary schools and enroll 
in postsecondary educational programs." (Federal Register/Vol. 47, No. 34, 
Subpart A, 643.1, Feb. 19, 1982). 

Stated objectives related to this goal are to identify qualified individuals with potential 
for postsecondary education, and encourage such individuals to complete high school and enroll in 
postsecondary education, provide them with information about financial aid, and develop their 
awareness of educational and career opportunities. 

Among the projects surveyed for this study, while the activities used to achieve these 
objectives were similar, the models for delivering services to students varied. All projects provided 
workshops encompassing information on postsecondary education options, careers and financial 
aid, how to select colleges and complete application forms, increasing self awareness, study skills 
and test taking. Nearly all projects also provided individualized counseling, application assistance, 
and field trips to college campuses, and staff also monitor students' academic progress. Helping 
students locate financial aid and successfully complete the aid application process to secure 
enough funding to meet their college costs was another common feature across projects. 

Within this broad framework, :he projects surveyed had somewhat different 
emphases, depending on the student population served and local circumstances. For example, 
projects serving primarily African-American or Latino students emphasized cultural awareness 
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more than projects serving racially diverse student groups. While most project directors 
interviewed considered postsecondary enrollment their primary goal, a few placed equal, or 
perhaps more, emphasis on dropout prevention and high school graduation. This was the case for 
several rural projects in areas with unusually high dropout rates where, if students did graduate 
from high school, there was a strong likelihood of their pursuing postsecondary studies. Projects 
also differed in the emphasis placed on career exploration, goal setting, and self-esteem building. 
Projects in geographic areas with large numbers of colleges (Ohio, Pennsylvania) seemed to 
emphasize making college choices more so than projects where the only options are nearby public 
institutions. 

A number of other differences were found among the projects contacted. Some 
differences appear to have little effect on service delivery. Whether or not a project was sponsored 
by a community agency or a higher education institution or whether it was associated with an 
Educational Opportunity Center, seemed to make little difference in how services were delivered. 
All projects emphasized individual counseling more in 11th and 12th grades and workshops and 
group activities with younger students. 

One difference that affected the service delivery model was whether the project was in 
an urban or rural setting. Urban projects tended to focus on a smaller number of target schools, 
serving larger numbers of students at each school. Rural projects served many more schools 
scattered throughout large geographic areas. The number of schools served by an urban project 
varied from 5 to 17. Rural projects served from 10 to 100 high schools. 

Urban projects typically had counselors working with target schools one to three days 
a week, while rural projects typically had counselors visiting schools once or twice a month. 
Because counselors in urban projects spent more time in a few schools, they tended to see more 
students individually than is the case for rural projects. In part, this is because urban projects 
encountered difficulties getting teachers to release students from classes for group activities. 
Rural projects, on the other hand, worked with students primarily in structured group situations 
during the regular school day. 

Another difference affecting student participation in urban and rural projects was 
competition from other programs. In rural areas, Talent Search was one of the few speaal 
activities offered in a school. In urban settings, by contrast, Taient Search projects competed for 
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student attention with early-release work programs, dropout prevention programs, and numerous 
other activities. Urban projects also faced problems with students frequently moving from one 
place to another, transferring to other schools, and erratic attendance. In rural projects, there 
seemed to be more stability in students' living situations, making it easier for project staff to follow 
them over time. 

Programs also differed in the degree of parental involvement. A few programs had 
frequent individual contact with parents, beginning when students were first recruited for the 
project and including home visits. Many projects had attempted parent workshops with varying 
degrees of success. Workshops generally were more effective with middle school parents, who are 
more actively involved with their children's education than high school parents. Mailings and 
telephone calls to parents at home, particularly in relation to the financial aid application process, 
also were common. 

Two other significant differences found among programs were the student/counselor 
ratios and the per-student cost. Student/counselor ratios varied from 200/450. Per-student costs 
in the projects contacted varied from $102 per student to S276 per student. Both urban and rural 
projects had high and low stuihnt counsel or ratios an 3 high and low per-student costs. 

Interestingly, postsecondary enrollment rates seemed to bear no relationship to 
project differences. Postsecondary enrollment rates among the 19 projects represented varied 
from 60 percent to 90 percent of the graduating seniors served. There were two urban and rural 
projects which placed 60 percent of their graduates, while two others placed 90 percent of their 
graduates. This raises the question of the extent to which factors such as frequency of student 
contact, emphasis on individual counseling versus group workshops, student/counselor ratios, and 
per-student cost affect postsecondary enrollment rates of Talent Search participants. 

Projects serving large numbers of students who are recent immigrants and whose first 
language is not English experienced some special problems. Project directors discussed at length 
the challenges of delivering services in multiple languages. These included translating publications 
into various languages, something they noted as an expensive proposition, and having counselors 
who can communicate with parents in their native languages. Such projects also faced the difficult 
problem of what to do with illegal aliens. According to federal regulations (Subpart A, 643.3), all 
project participants must be U.S. citizens, permanent residents of the U.S.. or otherwise eligible 
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non-citizens according to the U.S. Immigration and Naturalization Service. Talent Search projects 
cannot serve illegal aliens. In target high schools, particularly Hispanic and Haitian populations, 
there are large numbers of illegal aliens. In such situations, staff found it awkward recruiting for 
Talent Search without adding to the sense of inadequacy of students who do not have legal status. 



3* Critical Components of Talent Search 

Project directors interviewed were asked to describe the critical components of Talent 
Search which contribute to the success of the program in enabling students to complete high school 
and enroll in postsecondary education. 

Critical components were described in several different ways. Some project directors 
talked about the ways Talent Search effects positive changes in student behavior, which result in 
students' furthering their education. Such changes include: 

■ increasing students* motivation to graduate from high school and pursue 
postsecondary education. Project directors felt that, without Talent Search, 
most of the students they serve would not receive encouragement to undertake 
higher education; 

■ increasing students' awareness of educational and career opportunities; 

■ increasing students' self-awareness and self-esteem. Developing students' 
confidence in their abilities to succeed in school. Developing in students a "can 
do" attitude; 

■ developing students' decision making skills; and 

■ developing students' capacity to take responsibility for themselves. 

Other project directors described the critical components of Talent Search in terms of 
program activities which they felt had the most impact on students' likelihood to further their 
education. Informational workshops on career exploration, college selection, and financial aid 
were often mentioned. Life skills, SAT preparation, and study skills workshops were also 
frequently noted, as was assistance with completing admissions and financial aid applications. 
One-on-one counseling with school and personal problems also was viewed as important, including 
academic advising. Field trips to college campuses and cultural activities were considered critical. 
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Another frequently mentioned component involved linking Talent Search participants to 
community resources, including social services and enrichment programs. 



Advocacy for students in resolving a wide variety of problems was also oftrn 
mentioned. Such problems involved teachers, family members and peers, and problems 
encountered in college admission and financial aid processes. 

Other factors considered important were staff members who served as role models for 
students, particularly African-American males, and activities to encourage parents to become 
actively involved in helping their children with education planning. Many people mentioned the 
importance of the relationship that students establish with Talent Search staff who, for many, are 
the first adults they have known who genuinely care that they go to college. 

4. Impact of Talent Search 

A major focus of the interviews with project directors was the impact Talent Search 
has on students, their families, the schools they attend, and their communities. Project directors 
had wide-ranging views on the various ways in which Talent Search has impacts upon these groups. 

4.1 High Schrol Graduation/Postsecondary Enrollment 

Everyone interviewed believes that Talent Search participation results in increased 
high school graduation and postsecondary enrollment rates for participating students than the 
rates for similar students who have not participated in Talent Search or other programs designed 
to increase their likelihood of going to college. 

42 Financial Aid 

People also believe that Talent Search students are more likely to receive financial aid 
than other students, and that the amount of financial aid they receive is likely to be greater than 
that received by similar students not involved in Talent Search. 
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People think that Talent Search participants are more likely to complete college 
admissions requirements than are other similar students. Specifically, students are more apt to 
have taken college preparatory courses and college entrance examinations than their counterparts 
not served by Talent Search. 

4.4 Postsecondary Retention 

There was considerable disagreement on whether Talent Search participation results 
in improved college retention rates for students served as compared with similar students. Some 
directors felt strongly that because Talent Search enables students to make more appropriate 
college choices, students are more likely to remain in college than they would be otherwise. 
Others felt that because the services provided by Talent Search are considerably less than those 
provided by Upward Bound or other intensive early intervention programs, Talent Search cannot 
be held accountable for students' success in college. They believe that while Talent Search can 
lead students to the college "gates", projects have little control over how students do once they're 
on campus. 

4.5 Academic Performance 

People also disagreed on whether Talent Search participation results in improved 
academic performance. Most people think that this is more likely for students participating in the 
recently launched middle school initiative, which focuses more on building academic skills than 
does the high school component. With regard to high school participants, some people think that 
if Talent Search successfully increases the motivation of students, this results in improved 
academic performance. Others believe that without the sustained academic interventions provided 
in the middle school initiative or in programs such as Upward Bound, it is unrealisric to expect 
significant improvements in students' academic performance. 

In addition to these outcomes, prcieJt directors identified a number of other areas in 
which Talent Search participation has a positive impact on students, parents, schools and 
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communities, as described beiow. These points, which are summarized below, are a composite of 
everything mentioned more than once. Some points surfaced frequently, while other were noted 
less often. 



Information 

There was widespread agreement that Talent Search participation increases 
knowledge of postsecondary education and career opportunities. Both students and parents have a 
much more detailed understanding of types of postsecondary institutions, sources of financial aid, 
admissions requirements, and admissions and financial aid application processes than they would 
have otherwise. Students have an increased understanding of career options and how to prepare 
for them as a result of Talent Search participation. Parents develop a better understanding of the 
policies and practices of the public schools that their children attend and how to relate effectively 
to teachers and other school staff. This is especially true for middle school parents. Guidance 
counselors and teachers at the target schools develop greater understanding of financial aid, 
admissions testing and special college programs for minority and other non-traditional students as 
a consequence of Talent Search. Many project directors interviewed indicated that, without Talent 
Search, students and counselors at the target schools simply would not have access to current 
information on higher education and financial aid. 



Attitudes/Aspirations 

People interviewed felt that Talent Search participation results in changes in student 
attitudes and aspirations. Students' perceptions of their future goals change. They are more likely 
to have high school graduation and college attendance as goals than similar students not involved 
in Talent Search, and to believe that they can be successful in college. They develop increased self- 
confidence and higher self-esteem as a result of Talent Search participation, and are more likely to 
have an "I can do it" attitude than they had before. Their motivation to do well academically 
increases as does their capacity to take responsibility for themselves and their actions. They 
display more positive attitudes towards schooling and are more likely to think of themselves as 
long-term learners. Talent Search participation also makes students more comfortable with the 
college decision making process than other similar students. 



People thought that parents' attitudes change significantly as a result of their 
children's involvement in Talent Search. Parents become less skeptical about the value of higher 
education and feel more positively about their children going to college than they had previously. 
They also develop beliefs that their children are talented and have greater confidence in the future 
possibilities for their children. They are more likely to believe that a college education is worth 
while for their children to pursue. 

There were divergent views on the extent to which Talent Search changes the 
attitudes of target school staff towards students. Some people felt that Talent Search results in 
positive changes in the perceptions of counselors and teachers about which students are "college 
material" and reduces the ethnic stereotyping of certain groups of students. People also thought 
that Talent Search results in increased awarene^ on the part of school scaff of the need to provide 
career and pre-college guidance to all students, not just those in the top 10 percent. Others 
observed that Talent Search has had little impact on helping schools recognize the potential of 
first-generation students and expressed disappointment in this fact. 



Application Processes 

Another positive impact of Talent Search identified by many people is the increased 
likelihood of students and parents completing admissions and financial aid application processes. 
Most project directors believed that, as a result of Talent Search, students and parents are more 
likely to complete college and financial aid applications than they would without the help the- 
program provides. They also thought that, because of the advocacy provided by Talent Search 
counselors, problems commonly encountered in the admissions and financial aid application 
processes are more likely to be resolved than they would have been without Talent Search 
intervention. People noted that Talent Search participation results in students applying to colleges 
they would not have considered without the program's assistance and encouragement. A few also 
thought that students are more likely to apply to four-year colleges than they would have without 
Talent Search. 
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Effects on Other Family Members 

Project directors identified several ways in which Talent Search affects family 
members of the students served. They thought that Talent Search participation often results in 
other children in a family going to college, especially if the student served by Talent Search 
receives an attractive financial aid package. People also thought that, in general, parents are more 
likely to encourage other children to consider higher education and more likely to be actively 
involved in helping their children plan tor the future. As a result of having a better understand ; ng 
of how the schools their children attend operate, people see parents as better equipped to 
participate in teacher conferences and to advocate for their children with school staff than they 
had seen before. Obviously, this helps net only the child participating in Talent Search but other 
children in the family as well. 



Quality of Counseling at Target Schools 

Most people interviewed believe that Talent Search results in improved college 
counseling at the target schools for mid-quartile students who otherwise would be unlikely to 
receive such guidance. This is true particularly for students whom counselors do not consider 
"college material" and who otherwise would be likely to fall through the cracks and not be 
encouraged to further their education. The presence of Talent Search in schools was also thought 
to demonstrate the need for additional assistance for special groups such as bilingual populations 
who otherwise would not have access to a bilingual counselor. 



Impact on the Target Communities 

In general, people did not have a well-defined sense of the impact that Talent Search 
has on the communities in which projects are located. They speculated that there is likely to be 
increased community awareness of the availability of financial aid and how to apply for it. They 
also thought that there might be somewhat increased community awareness of college 
opportunities for low-income students. People serving rural communities observed that Talent 
Search H;. S pels widely held myths that college degrees have little worth. Rural communities served 
often point to Talent Search graduates who have gone to college and then on to successful careers. 
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With regard to youth organizations in the target communities, there was a sense that Talent Search 
results in increased involvement of youth leaders in encouraging students to plan for the future 
and to consider higher education. 



Impact on Postsecondary Institutions 

Several people mentioned ways in which they thought contact with project staff and 
students affects the postsecondary institutions that Talent Search students attend. Several cited 
instances in which colleges modified their admissions and financial aid policies in order to be more 
responsive to minority and bilingual students as a result of discussions with Talent Search staff. 
Contact with Talent Search staff also increases the understanding of admissions and financial aid 
officers of the needs of low-income and minority students, and sometimes results in the provision 
of improved student support services for such students. For example, several institutions planned 
special orientation workshops for Talent Search graduates in the summer prior to their freshman 
year. Talent Search also affects colleges by serving as a training ground for young professionals, 
especially minorities, for positions in higher education administration. Several project directors 
mentioned staff members who had moved on to positions as financial aid officers or career 
counselors in college placement offices. It was felt that without their experience in Talent Search, 
such individuals otherwise might not have had opportunities to prepare for careers as higher 
education professionals. 

Another outcome of Talent Search has been the establishment of networks of 
organizations and groups concerned with increasing access to higher education for low-income and 
minority students in general. Such groups consist of college representatives, high school staff, and 
community agency personnel working together to increase educational access and reach out to 
students with information and assistance. 



5. How to Evaluate Talent Search 

The project directors interviewed identified various approaches for evaluating Talent 
Search and documenting program effectiveness. These approaches involve veriiying outcomes, 
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interviewing students and target school staff, survey instruments, pre- and post-tests, longitudinal 
studies beginning with younger students, and success stories. 

People thought it would be important to verify the outcome information included on 
the annual performance reports submitted by projects to the U.S. Department of Education, 
Specific types of data collected are listed in Attachment A. Information that needs to be verified 
includes high school graduation and postsecondary enrollment rates. 

People suggested comparing such rates with those of similar students not served by 
Talent Search, either at other high schools in the region where projects being evaluated are 
located, or with a national study such as the National Education Longitudinal Study. Some people 
also suggested that the current rates be compared to rates for the same high school prior to Talent 
Search services being available. Because some Talent Search projects have been in existence more 
than 20 years, this approach would be less likely to produce reliable findings. People also 
suggested reviewing high school transcripts and school attendance records of Talent Search 
participants to verify improvements in grades and attendance and whether students have taken 
college preparatory courses. 

Most people interviewed thought it would be useful to conduct interviews both with 
former students served by Talent Search and with counselors and teachers at the schools, either on 
an individual basis or in focus groups. Interviews with former students might include questions to 
determine the congruence between student expectations of Talent Search and how they were 
helped, the extent to which students got the information and assistance they wanted, and what 
difference this made in terms of what they did after high school. People thought that both students 
who enrolled in postsecondary education and those who did not should be interviewed. It was also 
suggested that students be asked how their behavior changed once they became involved in Talent 
Search, particularly with regard to courses they took, their attitude toward school and their 
involvement in extracurriculai activities. 

Interviews with counselors and teachers at the target schools could focus on their 
perceptions of how Talent Search made a difference for students in terms of attitudes, behavior, 
and achievement. It was strongly recommended that teacher? and counselors selected for 
interviews be identified by Talent Search staff. It was also suggested that staff of youth 
organizations be interviewed. 
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Some people thought that it might be useful to survey a random sample of Talent 
Search graduates, as well as counselors in the target schools and parents of Talent Search 
participants. The survey could ask people to rank the importance of particular activities in 
achieving the goal of students er, oiling in higher education. One of the individuals interviewed 
had done such a survey and produced interesting findings. 

Pre- and post-tests were seen as a way to measure increases in the knowledge of 
students and parents regarding educational postsecondary opportunities, careers, and financial aid. 

The possibility of a longitudinal study was discussed with regard to students starting 
Talent Search in the middle school component. It was thought that such a study would provide an 
excellent opportunity to observe changes in attitudes, behaviors, and performance over time. 

Several people thought it would be useful to develop profiles of Talent Search 
students who have experienced considerable success as a result of their program participation. A 
variation of this idea, which was not suggested but might produce interesting insights, would be a 
comparative case study of 10 - 12 Talent Search graduates, some of whom were successful and 
others who were not. Such a case study might compare students in terms of family history, school 
history, peer relationships, and what they did after high school. 

Many people expressed concern that, in contacting former students, care be given to 
identifying the program in the same way students had identified it. While some Talent Search 
projects are known to students as "Talent Search," others have different names. .Also, students 
whose primary contact with Talent Search counselors is at the target school may not have 
identified the Talent Search counselor as someone different from a regular school guidance 
counselor. 



6. What People Want to Learn From a National Evaluation 

When asked what they would like to learn from a national evaluation of Talent 
Search, people expressed interest in a number of areas. Ever>-one hopes for validation that Talent 
Search makes a difference in students' prospects for higher education and long term success. They 
are also interested in whether particular program characteristics make a difference in the broad 
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outcomes that are achieved, what are the most effective program strategies, and what changes 
might be made to improve the likelihood of future Talent Search participants enrolling in and 
succeeding in postsecondary education. 

The questions which have the greatest interest for the people interviewed relate to the 
effectiveness of Talent Search in achieving the program s overall goals. People want to know if 
Talent Search increases the high school graduation rates and postsecondary enrollment rates of 
students served and if students receive adequate financial aid. They are also interested in knowing 
what types of colleges participants attend and if they are more likely to attend four-year colleges 
than similar students who were not served by Talent Search. Some people are also interested in 
the types of majors that students select. Since career exploration is an important component of 
many projects, people want to know whether the college majors of Talent Search graduates 
represent a greater diversity of career areas than would be the case for similar students who did 
not participate in Talent Search. 

While the majority of people interviewed believe that the success of Talent Search 
should not be measured in terms of the retention of participants in college, everyone is interested 
in learning about the college experience of Talent Search graduates as compared to the experience 
of other students. People would like to know about the experiences of both students who 
successfully complete college and students who leave college before completing degrees. In the 
case of the latter group, people expressed interest in learning what factors contribute to students 
leaving college before degree completion, and particularly, to what extent financial aid plays a role 
in students leaving. People would also like the national evaluation to document the impact, if any, 
that Talent Search has on the families of participants. 

Considerable interest was expressed in learning about what it is that Talent Search 
does that actually makes a difference. People are interested in whether specific program 
characteristics make a difference in the likelihood of students enrolling in higher education. For 
instance, is there a direct relationship between particular program features and project success? 
Program features that people think may make a difference include student/counselor ratios, the 
dollar resources per student, the frequency of student contact, and the length of student 
participation in the program. 



Many people expressed interest in knowing whether there are some program activities 
that make more difference than others in terms of overall impact. For instance, do financial aid 
counseling and academic advising make a greater difference than field trips or study skills 
workshops? Likewise, people want to know what works and what doesn't work in terms of specific 
program objectives. Particular interest was expressed in activities which are effective in motivating 
students to go to college, getting parents involved, and improving student academic preparation. 
People also hope that the national evaluation will identify new and innovative program practices 
that are effective in reducing high school dropout rates and educating younger students and their 
parents regarding the financial aid process and college costs. 

Considerable interest was expressed in what changes could be made in Talent Search 
to improve future program effectiveness. People are curious to see if a more clearly defined 
model of an effective Talent Search program might emerge from the evaluation findings. They 
also are interested in knowing whether certain program adjustments would improve outcomes. 
For example, if the student/counselor ratio was reduced from 350/1 to 250/1, or if programs 
started working with most students in sixth grade rather than ninth grade, would there be a greater 
improvement in high school graduation and postsecondary enrollment rates? 

7. Recommendations 

Based on the study findings, the following recommendations regarding a national 
evaluation are offered: 

1. It might be useful to include several experienced Talent Search project 
directors on the advisory committee for a national evaluation. Several of the 
project directors interviewed have given considerable thought to program 
evaluation, and many have worked diligently over the years to determine how 
they can make their programs more effective. Their insights might be quite 
useful in developing and reviewing the evaluation design. 

2. The question of what accounts for the 30-point spread in postsecondary 
enrollment rates of the programs surveyed should be considered. While the 
projects chosen for this study did not represent a random sample, it was 
surprising to find no obvious correlation between program characteristics and 
postsecondary enrollment rates. It would be useful to understand what factors 
contribute to this substantia! difference in placement rates. 
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Based on the interviews conducted, it appears that the environment in that a 
Talent Search project operates significantly affects the usefulness of particular 
activities or program strategies. For example, it appears that the complexities 
found in an urban environment make certain activities which are highly 
successful in a rural environment less so for urban students. A national 
evaluation might look specifically at such issues. 
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Attachment A 

PARTICIPANT DATA COLLECTED BY TALENT SEARCH PROJECTS 



The following information is based on a review of the forms from 13 of the projects contacted for 
this paper. 

Participant Characteristics 

A. Personal Information 
Sex 

Age/Birthdate 

Race/Ethnicity 

Social Security Number 

Physically Handicapped 

Family Size, Number Living at Home 

Family Income: Amount and source 

Citizenship Status 

Language Spoken at Home 

Parents* College Education Level (4-year degree or not) 
Parents' Marital Status 

B. School/ Academic 

Favorite High School Subject 

Weakest High School Subject 

Expected High School Graduation Date 

School Transcripts 

Extracurricular Activities 

High School/Middle School Attending 

SAT/ ACT Scores 

Colleges Attended 

Current Grade Level 

C. Future Plan 

Career Goals 
Educational Plans 
Planned College Major 
Employment Status 

D. Talent Search Participation 

Serv ices Requested 
High School Courses Taken 
Participant Needs Assessment 
Academic Work Plan 
Teacher Recommendation 
Counselor Recommendation 
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Services Provided to Participants 

Counselor Contact Logs: Maintained for each student 

Monthly Counselor Tally Sheet: Summarizes total number of students served in various 
categories 

Target School Participant Log: List of participants and type of services provided to each 

Student Service Record Sheet/Progress Report 

■ Maintained for each participant 

■ Summarizes services provided 



Participant Outcomes 

Participant College Application Summary Sheet 

High School Achievement/Graduation 

■ Transcripts 

■ Class rank by year 

■ Admissions test scores 

Postsecondary Enrollment Status 

■ Self-reported 

■ Verified by postsecondary institution 

Financial Aid Award (type and amount) 

■ Self-reported 

■ Verified by postsecondary institution 



Evaluation Data 

A. Target Schools 

Teacher Evaluation Forms 
Tutor Evaluation Form 
Principal Evaluation Form 

B. Participants 

College Trip Evaluation 
Tutor Evaluation Form 
Program Evaluation Form 
Workshop Evaluation Form 
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1. Introduction 

The focus of this paper is on the identification of appropriate outcome measures 
related to the missions and goals of the Talent Search program. It is believed that the emphasis of 
future assessment efforts should be on the outcomes associated with such a program, rather than 
in the operation of the program itself, as was the case in the 1975 evaluation of the Talent Search 
program (RTI, 1975). Outcomes assessment, instead, is recommended as the main mechanism to 
evaluate program^ effectiveness (Banta, 1988). The outcomes assessment approach stresses the 
products associated with a program as opposed to the specific intervention strategies identified 
with the program (Ewell, 1985). If assessment efforts emphasize the intervention strategies 
themselves, it is possible to lose sight of what really constitutes program effectiveness. Stressing 
outcomes, on the other hand, induces the agency to appropriately revise and articulate 
intervention strategies. This paper builds upon the proposition that the identification of such 
outcomes should be driven, on the one hand, by what is known about the college choice process 
(Hossler, Braxton, & Coopersmith, 1988) and, on the other hand, by the missions and goals of the 
Talent Search program. In this context, outcomes assessment for this program should involve the 
systematic process of collecting information documenting the extent to which the Talent Search 
program (1) brought about cognitive and affective changes impacting on the development of 
student educational aspirations and abilities, (2) enabled the student to appropriately search for 
and choose an appropriate institution to attend, and (3) enabled students and their parents to 
secure all information pertinent to college attendance (e.g., financial aid). 

By looking at what students and graduates know and are able to do with knowledge 
acquired, as well as their perceptions of the quality o f the program and services, policy makers and 
analysts can obtain important information about the program's ability to meet stated goals. It is 
further believed that the impact of such a low-intensity program on the family, the institution, and 
the community would be emp'rically difficult to assess beyond the intended scope of the program 
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itself. Moreover, while the mission of the program embraces both traditional and non-traditional 
(adult) student populations, the focus of this paper rests on the traditional student. One reason 
for targeting this student population is that the bulk of federal funding is primarily directed 
towards elementary, junior, and high school students. Secondly, and more importantly, substantial 
theoretical and empirical research exists as to the process and determinants of college choice for 
this group (e.g., Hossler, Braxton, & Coopersmith, 1988; Hossler & Gallagher, 1987; Stage & 
Hossler, 1989; Chapman, 1981; Ekstrom, 1985: Manski & Wise, 1983; Sewell & Shah, 1978; 
Hauser & Anderson, 1991). 

2. A Theoretical Framework Guiding Student College Choice 

The model advanced by Hossler, Braxton, and Coopersmith (1988) provides the basis 
for a theoretical framework to examine student college choice and for identifying relevant outcome 
measures. College choice is seen as a complex process involving four interrelated stages. These 
four stages include: (1) development of predispositions to attend college or educational 
aspirations, (2) a student's search for potential institutions to attend, (3) the choice among 
alternative institutions, and (4) the actual enrollment in an institution of higher education. In this 
context, Talent Search programs are seen as affecting each single stage. Each stage is associated 
with a specific age cohort that corresponds to grades 6 through 12. Furthermore, each phase has 
particular outcomes that manifest the cognitive and affective development of the student. These 
four stages, cumulatively., prepare the high schoc 1 student to pursue a college education (see Table 
1 at the end of this paper). The model also presumes that each phase is the product of particular 
determinants or factors. The factors listed for each stage in Table 1 reflect those identified in the 
literature as having the strongest effects. Finally, the model specifies that Talent Search program 
goals address each single stage in the student college choice process. 

2.1 Predisposition Phase 

This phase involves the development of occupational and educational aspirations 
(Hossler, Braxton, & Coopersmith, 1988; Hossler & Gallagher. 1987). In other words, the high 
school student comes to value education and develops the desire to attend college, The 
predisposition stage begins as early as the sixth or seventh grade, and by the end of the ninth 
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grade, most students have already developed occupational and educational aspirations (Ekstrom, 
1981). The literature suggests that the main determinants of the predisposition stage are: 

(1) the student's family income or socioeconomic status (SES); 

(2) parental levels of education (e.g., father's education, mother's education, and 
combined level of parent's eduction); 

(3) student ability as measured by GPA, academic test scores, academic standing, 
and high school grades; 

(4) parental encouragement and support to attend college; 

(5) encouragement and support by peers; and 

(6) whether or not the student is enrolled in an academic track in high school. 

Research has indicated that there is a complex process whereby these factors interact 
among themselves in shaping student predispositions to attend college. For instance, Hossier, 
Braxton, and Coopersmith's (1988) review of the literature indicates that the role of SES indirectly 
affects how much encouragement parents provide the student and the student's academic ability. 
Parental encouragement, however, has consistently been found to be the strongest predictor of 
students' educational aspirations (Stage & Hossler, 1989). 

Outcome Measures of the Predisposition. Stage 

The following outcomes are appropriate measures associated with the predisposition 
phase These outcome measures include: (1) math, writing, reading, and critical thinking skills 
(student ability); (2) perceptions and attitudes regarding career or occupational aspirations; 
(3) perceptions and attitudes regarding the student's educational aspirations; and (4) the student's 
enrollment and participation in a college-bound curriculum. 

These specific outcome measures represent quantifiable variables that can be used in 
assessing the effectiveness of a Talent Search program in relation to the first phase. Some 
examples of intervention strategies used by Talent Search programs that are related to the 
predisposition phase include: (1) career exploration and decision-making workshops: 
(2) academic tutorial sessions; (3) visitations to postseeondary educational institutions: 
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(4) assessing academic potential through standardized tests; (5) self-esteem and self-concept 
workshops; and (6) field trips to workplaces for different occupations. 



22 Search Phase 

The second phase in the college choice process involves the accumulation and 
assimilation of information necessary to select institutions for o final choice set. The stage begins 
at the point where a student has developed firm educational aspirations and has embraced the idea 
of attending college. Specific to this phase are activities that engage the student in attentive, 
active, and interactive endeavors to gather information about potential institutions that may be 
considered for enrollment from a number of possibilities. It is at this stage that students begin to 
actively interact with potential institutions. Visiting campuses, securing catalogs, and talking to 
friends about college are some of the activities used in seeking such information (Hossler, Braxton 
& Coopersmith, 1988). Moreover, it is believed that the communication strategies used by 
different institutions to attract students to their respective campuses is extremely varied and that 
the quality of the searching process by students and their families varies according to their 
socioeconomic status (Hossier & Gallagher, 19S7), The student's choice set consists of a group of 
colleges and universities that he or she has decided to consider and seek information about in 
order to make a better final matriculation decision. 

The search phase not only begins at the time that a student acquires a desire to attend 
college, but has also been found to be associated with a specific grade and age cohort. This stage 
usually begins by the 10th grade and continues during the 11th grade and, at times, on through to 
the middle of the 12th grade. 

Four factors have been found to directly affect the search stage in the college choice 

process: 

(1) the acquisition of established educational aspirations by students; 

(2) the identification of occupational plans by high school students wanting to 
attend college; 

(3) the student's family income or socioeconomic status (SES); and 
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(4) the saliency of potential institutions. 



The degree of sophistication reflected in a student's search is also related to (5) 
student ability. Those students with higher ability (as measured by standardized test scores) and 
parental income are more likely to expand both the geographical range and the quality of the 
institutions that they consider (Hossler & Gallagher, 1987). Moreover, those students who do not 
possess those abilities and do not come from higher socioeconomic levels have a tendency to rely 
almost exclusively on high school counselors or such informal sources as friend: for advice. It is 
because of these circumstances associated with the search stage that assessment of Talent Search 
programs must focus on their efforts in making students and parents aware of the diversity of 
postsecondary options that are available to them and thereby affecting the choice set established 
by the student. 



Outcome Measures of the Search Phase 

Three outcomes are associated with the search phase of the college choice process: 
(1) the choice set or the listing of tentative institutions by students wanting to attend college; (2) 
narrowing the list of institutions; and (3) gathering information on each institution in the choice 
set. 

Examples of intervention strategies that address the search process by the Talent 
Search program include: (1) counseling to provide assistance to students in narrowing their 
postsecondary programs and institutional choices: (2) collecting and disseminating information 
regarding the variety of postsecondary educational opportunities: (3) providing academic tutorial 
sessions; and (4) participation in "College Days" activities. 



23 Choice Phase 

While the search phase is exploratory in nature, the choice stage is one of evaluating 
the potential postsecondary alternatives and initiating the application process (Hossler & 
Gallagher, 19S7; Hossler, Braxton, & Coopersmith. 1988). Students at this stage are engaged in 
synthesizing information about institutional attributes, developing strong preferences about 
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particular institutions, evaluating their academic abilities in relation to admission requirements, 
and pondering about their choices for financing their college education. A characteristic of the 
choice phase is the dynamic nature of the evaluation process engaged by both the student and his 
or her family. Perceptions regarding institutional characteristics among those institutions 
identified in the choice set are continuously expanded and, subsequently, preferences among the 
colleges and universities chosen may vary from one moment to another. Acquisition of additional 
information from catalogs, counselors, friends, relatives, and agencies add to the cumulative 
information from which the student can evaluate his final choice. A matching process between the 
student's academic abilities, his potential to meet tuition costs, the proximity of the institution 
selected, and other institutional attributes and the student's final selection exemplifies this stage. 
What most characterizes the choice phase is the role of perceptions in shaping the evaluative 
processes. The quality of the information acquired plays a key role as to whether or not the 
evaluation process leads to correct decisions. 

Factors which have been found to be determinants of the choice stage include: 

(1) the student's educational and occupational aspirations; 

(2) family income and socioeconomic status (SES); 

(3) student academic ability; 

(4) parental educational attainment; 

(5) the degree of parental encouragement and support towards particular 
institutions; 

(6) students 1 perceived institutional attributes such as the quality and prestige of 
the institution, campus life, major field of study, geographic location; and 

(7) the students' perceived ability to pay educational costs. 

It has been suggested that both SES and student ability are positively associated with 
the selection of more prestigious institutions with higher admissions standards. As the 
socioeconomic status and/or scholastic aptitude of the student increases, the more likely it is that 
he or she will apply to more prestigious institutions. Likewise, financial considerations (e.g., cost 
of attending a;id financial aid) have been found to be associated with the selection of institutions 
and with the submission of applications to institutions. H:\vever. the strongest determinants of the 




choice phase are those factors related to parental expectations, SES, and student ability (Hossler, 
Braxton, & Coopersmith, 1988). 



Outcome Measures of the Choice Phase 

Six outcome measures are associated with the choice stage: (1) awareness of 
institutional attributes and admission criteria; (2) attaining scholastic attitudes and aptitudes 
necessary to attend a particular institution; (3) perceived support from family and friends to attend 
a particular institution; (4) commitments to attend a particular institution; (5) awareness of college 
expenses and financial aid; and (6) submission of applications. 

Some examples of intervention strategies employed by Talent Search programs to 
produce those outcomes associated with the choice phase are: (1) counseling assistance in filling 
out admissions, financial aid, and application forms; (2) collecting and disseminating information 
regarding student financial aid and academic assistance; (3) tutorial sessions and summer 
programs to enhance student academic ability; and (4) college orientation workshops. 

2.4 Enrollment 

Although the literature typically identifies enrollment as a characteristic of the choice 
phase (Hossler & Gallagher, 1987; Stage & Hossler, 1989), it is important to make a distinction 
between the selection of the final institutions to which students apply to and actual enrollment. 
While in the choice stage, students develop strong dispositions to attend a particular institution, 
the enrollment phase is characterized by the final selection of an institution and actual attendance. 

The literature is consistent as to what factors most affect participation rates (Hossler, 
Braxton, & Coopersmith, 1988; St. John, 1991: Jackson, 1988). These factors include- (1) 
increased aspirations for higher levels of educational attainment; (2) family income and 
socioeconomic status (SES): (3) improved academic preparation in elementary and high school: 
(4) parental educational attainment; (5) the degree of parental encouragement and support toward 
particular institutions; (6) a student's commitment to a particular institution: (7) student 
perceptions of net costs; and (8) financial aid a\ailability at the selected institution. 
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Studies have found that not only is the reception of financial aid positively related to 
enrollment rates (Jackson, 1988; St. John, 1990 a,b; St. John & NoelL 1989) but that perceptions 
associated with financial aid offers are as likely to affect the participation rates of students (St. 
John, 1991), Financial considerations have an impact on decisions as to whether or not to attend 
college and what institution to attend (Porter, 1991). 

Outcome Measures of the Enrollment Phase 

Three outcome measures are associated with the enrollment stage: (1) 
preregistration; (2) applying for financial aid; and (3) enrollment and attendance in the selected 
institution. Some examples of interventions that Talent Search programs have used that address 
the enrollment stage include: (1) counseling in filling out admissions, financial aid, and application 
forms; and (2) checking on the status of admission, financial aid, and actual enrollment of students. 

3, Assessment Issues 

This paper is based on the proposition that the assessment of Talent Search programs 
should begin with a set of assumptions regarding the objective of the program, that is, college 
choice. It is presumed that college choice itself is a complex process comprised of four stages and 
that each stage has unique outcomes and age cohorts associated with each phase. Furthermore, 
because of these two characteristics, it is suggested that the assessment of Talent Search programs 
be driven by the unique nature of the outcomes associated with each phase. It is also believed that 
a Talent Search program cannot be evaluated as a single effort. Rather, the assessment of Talent 
Search should be viewed as a series of evaluations directed at different subcomponents that 
attempt to address different stages in the college choice process. 

In the following discussion, we suggest who the most appropriate evaluation 
participants might be in assessing Talent Search programs, identify possible data collection 
approaches, and discuss methods of analyzing datu related to Talent Search. The last column of 
Table 1 (at the end of this paper) parallels the discussion below 



3.1 



Participants and Respondents in a Talent Search Evaluation 



Previous evaluation efforts have focused on the delivery mechanism under the 
assumption that such an evaluation approach would lead to improvement of the delivery system 
with subsequent spill-over effects on attaining the outcomes (RTI, 1975; Franklin, 1985). This 
approach, however, totally disregards the recipient of such services by not acquiring pertinent 
information as to whether or not the program meets its objectives (Ewell, 1991). Both approaches 
should be used in evaluating the effectiveness and success of the Talent Search program, but the 
assessment of outcomes should be stressed. 

Although it is important to secure tallies on the number of students receiving services, 
using frequencies of participation rates, retention rates, the number of students who actually 
enroll, and opinions from administrators and staff, this information merely reflects a static and 
one-sided view of the program. It is more important to obtain information on the development of 
student attitudes and behaviors that research indicates are good predictors of college choice and 
college attendance. This information can be enriched by surveying parents, local high school 
counselors and administrators, and college counselors and recruiters regarding their attitudes of 
the Talent Search program and services. This strategy has been pointed out as the most dynamic 
and comprehensive component of the evaluation process (Ewell, 1991). 



Methods for Developing Databases 

Appropriate Talent Search databases should not rely only on ad hoc questionnaires. 
Those in charge of developing databases for future evaluation purposes should consider 
developing or employing instruments that evolve out of theory, whose validity and reliability can be 
well-documented, and that additionally tap critical attitudinal, cognitive, effective, and behavioral 
measures related to the college choice process. This approach will enable comparisons across the 
different phases and modalities of the subcomponents of the Talent Search program. Data 
collection should also parallel specific periods when programmatic and personal characteristics are 
most likely to exhibit their strongest effects on college choice (Cabrera. Stampen. & Hansen, 
1990). Annually collected data will also help to identify short-term effects, as well as recurring 
effects, among motivational and ability variables. Researchers bhould also consider developing 



additional sources of information that will enable them to validate the extent to which the program 
(interventions) had the desired effects (e.g., college transcripts, high school transcripts). 

Two modes of data collection appear appropriate for collecting information on outcome 
measures identified in Table 1. Cross-sectional data collection strategies can be employed to 
provide a profile of both student and program characteristics, identify the extent to which the 
intervention strategies reach the target population, document current levels of satisfaction with 
particular intervention strategies, and collect retrospective information from recipients regarding 
the quality of the services, availability of services, and the degree of engagement in program 
activities. 

On the other hand, longitudinal databases following specific cohorts over time can be 
used to assess the carry-over effects of factors from one phase to another. These databases can 
help to document the extent to which the intervention strategies bring about changes in students' 
educational and occupational aspirations, academic progress, and awareness of pertinent financial 
and institutional options. In addition, longitudinal databases can be used to estimate probabilities 
describing student movements among intervention strategies aimed at the four phases of the 
college choice process. The databases can also be merged with future databases tracking ex- 
recipients of Talent Search services throughout their postsecondary experiences. Such databases 
that capture college enrollment behavior, transfer, stopout behavior, and persistence to graduation 
can be used to document the extent to which particular types of programmatic intervention 
strategies are more likely to yield satisfactory results. 

Subsequently, the longitudinal approach in de\ eloping databases can answer 
important questions such as: Does the same cohort make progress at each phase of the college 
choice process? Do recipients take advantage of different services? Do these students graduate 
from high school? Do recipients actually enroll in higher education institutions? Once enrolled, do 
they receive appropriate financial aid awards? And do recipients persist to graduation in college? 
Without information on such basic questions, one cannot make valid judgements about the short- 
term and long-term effectiveness of the Talent Search program. 
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Research Paradigms 



There are two research paradigms and numerous data analytic procedures that can be 
used in assessing the Talent Search program. The use of one paradigm over another should be 
dictated by the nature of the research questions and the amount of knowledge in the literature 
about particular outcomes. 

32.1 Quantitative Techniques 

Several statistical techniques are available to assess the Talent Search program. 
Markov analysis (Heneman & Sandver, 1977), for instance, is a statistical technique that can 
document the movement of a cohort across different phases at different times. Markov analysis 
relies on estimates of the probability that a particular cohort will engage in specific behaviors 
associated with each phase of the college choice process. Specifically, Markov analysis can be 
employed to estimate the likelihood of a Talent Search student cohort to: (a) participate in specific 
intervention strategies, (b) develop college going aspirations, (c) apply for college admissions, and 
(d) enroll in an institution of higher education. In order to conduct a Markov analysis of the 
Talent Search program, a longitudinal database that follows the oame cohort over time is needed. 
Relevant data measures would include: (a) degree aspirations for the cohort, (b) information as to 
whether or not students participated in the intervention strategic (c) whether or not the cohort 
applied to college, and finally, (d) information as to whether or not they enrolled in college. 
Markov analysis can be used to assess the extent to which different intervention strategies allow 
the students to make transitions from one phase to another. Furthermore, it can be employed to 
investigate the relative effectiveness of several intervention strategies in attaining the desired 
outcomes. This technique has been utilized in the past in documenting the effectiveness of the 
targeting of financial aid (Stampen & Cabrera, 1988). 

When group differences are desired, analysis of variance (ANOVA^ and multivariate 
analysis of variance (MANOVA) (Marascuilo & Levin. 1983) can be utilized. These techniques 
are particularly relevant in assessing differences between participants and nonpanicipants, and in 
controlling for extraneous factors to the Talent Search programs that can affect :he assessment of 
the outcomes. For instance, in the comparison of urban versus rural Talent Search programs, 
these techniques can remove the effect that a: fluency of the school district hn< jn the outcome. 
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Therefore, it is possible to document the net ct of the program on the outcome under 
consideration. 

When the focus of assessment is that of documenting the underlying structural 
patterns embedded among the different factors associated with outcome measures of the Talent 
Search program, causal modelling and multiple regression can be employed. These techniques can 
also allow the evaluator to investigate the direct effects of factors in the college choice process and 
the indirect nature of factors affecting final choice. Stage and Hossler (1989), for instance, used 
causal modelling to determine the patterns involving parental expectations, parental education, 
and other factors on educational aspirations among ninth graders. Because college choice has 
received little theoretical and empirical research, causal modelling remains a powerful alternative 
to discover the interrelationships among factors underscoring the different outcomes of the Talent 
Search program. Conceptual frameworks can help the evaluator select variables to use in causal 
modelling. Table 1 (found at the end of this paper) displays those variables found to exert the 
strongest effects on each phase of the college choice process and can be used in such efforts. 
Having established the underlying structural patterns through causal modelling, the evaluator will 
be in the position of offering explanations for particular outcomes such as college choice and 
enrollment. This knowledge can assist intervention strategists by focusing their attention on key 
variables and key relationships among them (Endo & Bittner, 19S5). 



3.2.2 Qualitative Techniques 

In areas of investigation where empirical research is still lacking (e.g., the search 
phase), a qualitative research paradigm may provide a more in-depth examination of the issue(s) 
at hand. Qualitative research may expand on factors not revealed in quantitative studies, while 
suggesting processes worth further inquiry (Attinasi & Nora, in press). There are several 
qualitative research methodologies that can be used: (1) naturalistic inquiries, (2) interviewing 
strategies, (3) critical incident techniques, (4) participatory observations, and (5) ethnographic 
studies (Guba & Lincoln, 1981; Lenning, 1988). 

Naturalistic inquiry is a qualitative strategy that involves having the researcher 
observe informants or participants in their natural setting. Data are collected through in-depth, 
open-ended interviewing of participating and neivarticipating students, and analyzed inductively 
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to generate grounded concepts for interpreting the context within which these students make 
college-going decisions. In the context of the Talent Search program, this could involve having a 
group of researchers witness how the program and interventions are actually conducted at 
different sites and interviewing recipients of the program concerning the experiences with said 
program. Critical incidence technique is a tool within the naturalistic inquiry paradigm that can 
assist the researcher in conducting such interviews. The focus of this technique is to reveal those 
critical components of the program that have brought about desired outcomes. It can also be 
employed to identify specific obstacles that inhibit the development of the same desired outcomes. 

Qualitative research may assist in the assessment of Talent Search programs in 
numerous ways. If an instrument is not available to measure desired outcomes, qualitative 
research techniques can be instrumental in the quan : t'ication of constructs not properly captured 
by standardized surveys. Qualitative research techniques can also be used to enrich the 
information uncovered by survey questionnaires and quantitative analyses. Naturalistic research 
has the potential for providing in-depth understanding of relationships between factors that are 
well established through the testing of causal models with survey data (Attinasi & Nora, in press). 
For example, although it may be established that a statistically significant relationship exists 
between particular quantitative measures of., say, parental encouragement and support and 
educational aspirations through causal model testing, it may be desirable to understand how the 
relationship plays itself out in the everyday lives of individual students; after all. there is only so 
much of humn behavior that can be ultimately captured in numbers. To achieve such 
understanding is the object of naturalistic inquiry. On the other hand, qualitative research 
techniques can also assist empirical assessments by guiding them as to potential relationships 
among variables not addressed in past evaluation efforts and in the formulation of hypotheses to 
be tested concerning intervention strategies. Integrating both qualitative and quantitative 
approaches may be useful in order to (1) address the issue of unexplained variance in quantitative 
models and (2) enhance understanding of relationships established between factors in causal 
models. 

Finally, the use of various methodologies in the assessment of the Talent Search 
program help to increase understanding of college choice, an issue that is complex not only 
because of the complicated nature of the choice phenomenon itself but also because of the 
additional factor of the cultural uniqueness o: the large number of minority recipients. Open- 
ended research techniques such as participant-observation ( Spradley. 19SC) and in-depth 
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interviewing (Stage, in press) allow the evaluator to assess culturally different people in an open- 
minded (but not emptied-headed) way. In summary, the complexity of potential issues argues for 
the use of a variety of methods, both naturalistic and quantitative, in the assessment of 
programmatic efforts. To understand college choice requires the development and testing of 
multi-factor quantitative models, and detailed understanding of student perceptions of influences 
on outcome measures, and findings from naturalistic approaches to illuminate the underlying 
associations identified in college choice models. 
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Review of Two Studies of Talent Search: 
The Research Triangle Institute's 1975 Study of Talent Search 
and Paul Franklin's 1985 Study for the College Board 

James E. Rosenbaum 



This report considers three issues: 

• Which policy questions are most important to attempt to answer through an 
evaluation of Talent Search? 

• Which program impacts are most important to examine and which are possible 
to measure through an evaluation? 

• Which research methods would make sense to use in evaluating Talent Search 
given the nature of the program and the questions we might want to answer? 

This report is organized in three main sections to address each of these questions. 
We examire these questions through a careful analysis of the results of two previous studies: The 
Research Triangle Institute's (RTI) 1975 Study of Talent Search and Paul Franklin's 1985 study for 
the College Board. The first was a large, multi-method study which produced an enormous five- 
volume report. The second was a smaller study which produced a 22-page report. Because of the 
greater detail and multiple methods of the former report, it will provide the greatest focus of this 
analysis. Unless otherwise noted, page numbers given refer mostly to Volume III of that report, a 
nine-chapter descriptive analysis of the program. 

L Which policy questions are most important to attempt to answer through an 

evaluation of Talent Search? 

The Talent Search program originated in the Higher Education Act of 1965 as a 
mechanism for identifying financially needy students and helping them to take 
advantage of the newly authorized Educational Opportunity Grant Program . . . 
Talent Search is designed to: 

a. identify youths of extreme financial and cultural need with an "exceptional 
potential w for post secondary' education and encourage them to complete 
secondary school and undertake further education; 
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b. publicize existing forms of student financial aid, including aid furnished under 
the Higher Education Act; and 

c. encourage secondary school or college dropouts of demonstrated aptitude to 
reenter educational programs (Vol. Ill, p. 1 of RTI report)." 

This program raises a number of fundamental policy questions. First, it is premised 
on an assumption that there are many youths of extreme financial and cultural need with an 
"exceptional potential 11 for postsecondary education. Given the great variety of institutions for 
postsecondary education in the U.S., and the great growth in higher education in the last several 
decades, this assumption is likely to be even more true today than it was in 1965. 

Second, it assumes that these youth need information and that publicizing existing 
forms of student financial aid can help youth attend postsecondary schools. As Franklin (p. 15) 
states: "low-income youth. . . do not know what middle-class people and college-educated families 
already take for granted about a college education: its advantages, its availability, its offerings, its 
requirements, its nature - in short, its potential value to them immediately and over time ... In 
particular, the disadvantaged often do not know about or understand financial aid programs, and 
fail to consider higher education because they assume they cannot afford it." Given the complexity 
of the college application process and 'he financial aid process, this assumption seems likely to be 
tnie. This suggests that an evaluation of Talent Search should examine wha t information and help 
youth need and the sources of information used bv various T alent Search projects. (This is 
discussed in detail in Section 3, "Useful Information for Projects.") Indeed, this is fundamental to 
the operation of Talent Search projects. For these projects to function, they must make some 
assessment of v. hat youth need to know and what kinds of help they need. An evaluation can help 
to verify their judgements on these questions and can also examine whether there are other unmet 
needs the project; could provide youth. 

Third, another assumption in setting up Talent Search is that the needs of these youth 
are not met by existing institutions. In particular, public high schools would seem the logical place 
for such services to be delivered. In designing Talent Search, the U.S. Congress decided that low- 
income youth were not being adequately served by the public schools. There was some basis for 
believing that in 1965, and there continues to be a basis for that belief in 1992. But the evidence 
exists only for isolated cases. There is no systematic information on what information and services 
are provided by public high schools to low-income youth with potential. Talent Search raises the 
policy question of what information and services are now being provided by public high schools. 
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why don't public schools provide the additional services that Talent Search projects are providing, 
and why do schools have difficulties in helping these students? While it cannot address this point 
completely, the evaluation should consider what public schools are doing in the areas where some 
Talent Search programs operate. 

Fourth, Talent Search raises the related question: what are the most important ways 
that Talent Search can supplement the services youth get from public schools? The aim of Talent 
Search should be to complement the schools' services, not to duplicate those services. The 
assessment of what schools are doing well and what they are not is required in order to define the 
services to be offered by a Talent Search project. This is likely to differ ill different communities. 

Fifth, are there some services that Talent Search could provide to schools that would 
increase the schools' capability and performance in helping youth? Given the fact that public high 
schools are already in place, and large numbers of youths are already there, policy makers must 
consider whether there are any steps that can be taken to improve the school's capability and 
performance in helping these youths. It is possible that the schools are beyond salvation and not 
worth helping, but that must be tested, not assumed. In contrast, it is possible that if school 
counselors were given certain information or services, then their effectiveness might dramatically 
improve. 

Sixth, the program's aims to "encourage them [youth] to complete secondary school 
and undertake further education...[and] to encourage secondary school or college dropouts of 
demonstrated aptitude to reenter educational programs" imply that Talent Search programs might 
provide not just information, but also academic tutoring and assistance. Again, instruction is part 
of the mission of the public schools, but they are clearly failing with large numbers of youth. Can a 
program like Talent Search provide such services and effectively improve youths' academic 
achievement? 

Seventh, Talent Search is designed to offer mostly information, advice, and counsel 
over a short period of time. Consequently, its effectiveness may be enhanced if it can enlist the 
help of others - church groups, parents, Big Brother and Big Sister programs, and other voluntee 
efforts. To what extent are Talent Search projects abie to enlist the help of others? 
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These are some of the general questions which an evaluation must address. We shall 
refer to many of these questions in the following sections. 



2* Which program impacts are most important to examine, and which are possible to 

measure through an evaluation? 

The RTI evaluation dealt with a multitude of important questions in assessing Talent 
Search projects. How do Talent Search projects define their goals, recruit and select clients, 
decide what help to offer, assess their outcomes and procedures, and assess what services to stress, 
which to improve, and which to decrease or drop? How do they interact with other institutions: 
local high schools, colleges, and other community groups? In the following sections, I will review 
the results of RTI's and Franklin's findings, and I will discuss which issues merit attention in the 
future evaluation. 

Before beginning, I will state an additional goal for an evaluation to consider. The 
next evaluation should be useful not only to national and regional offices, but also to project 
directors and staff. Evaluations often ignore the needs of project directors and staff. Focusing on 
outcomes, and ignoring project-level concerns, evaluations often reach summary judgements of 
overall effects, while producing little information that can help project directors and staff know 
how they are doing, what they could do better, or what additional steps they could take. 

This is unfortunate for two reasons. First, the evaluation can be more useful if it 
addresses issues of concern to projects. Evaluations that help project directors and staff can 
improve programs at least as much as evaluations that help central office staff. Second, 
evaluations which include the interests of project directors and staff improve their involvement in 
the information gathering process, and so they produce better information. 

Evaluation can focus on outcomes and process. While evaluations traditionally stress 
outcomes at the expense of process, evaluations of process can be extremely useful for improving 
projects. Such information can help central office staff understand what is going on in the local 
projects, and it can help various local projects know what other projects are doing. This can give 
them ideas about what they might try, and also let them know what activities are giving difficulties 
to other projects. 



9 

ERLC 



106 



1U3 



Both previous evaluations considered both outcomes and processes. My review of 
those evaluations will consider both topics and their implications for the next evaluation. 

Goals of Local Projects 

RTT reported five main goals of all Talent Search programs they visited: 

1. To provide general college and financial aid information 

2. To assist in making applications to postsecondary facilities (including help with 
PCS forms and taking aptitude tests) 

3. To assist with applications for financial aid (including fee waivers) 

4. To encourage clients and provide personal counsel 

5. To refer to other agencies for various types of assistance. 

These goals are consistent with the guidelines for Talent Search, and they were stated 
by all but one of the 20 sites visited. That site stressed completion of high school, giving secondary 
importance to postsecondary education. 

Yet programs differed in their emphasis. Some emphasized individuals' commitment 
to applying to postsecondary education, some emphasized actual applications, some emphasized 
placement, and some emphasized follow-up in postsecondary education to help with general 
adjustment. The RTI report also indicated a number of other specialized goals and special client 
groups (e.g., high school dropouts, jail inmates, etc.) (2.47). 

These goals all seem appropriate to the program's aims, so they do not need extensive 
evaluation. However, the evaluation might investigate why particular programs chose to 
emphasize particular goals or client groups. This may indicate areas of particular importance to 
the community or areas of special competence or interest to the staff or advisory board, or it may 
indicate idiosyncracies in the evolution of the local program. There may not be better or worse 
decisions on goals, but bv asking the questions, the evalu ation mav help the project directors and 
staff consider what they are doing and whv. Moreover, these different goals and emphases may 
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help the evaluation understand why a particular program does well at one outcome, but not at 
another. 



Franklin's 1985 study noted increased differentiation and greater clarity of mission as 
compared with the 1975 study (p.6). He noted that one project took a more developmental 
perspective, which influenced the recruitment, selection, and program offerings. The evolution of 
goals is a particularly interesting topic for learning from the projects' experiences. 



Recruitment and Selection of Clients 

RTI reports a variety of approaches to recruitment. Seven projects emphasized 
recruitment in high schools, 11 used high schools as well as other agencies (U.S. Employment 
Service, welfare programs, youth groups, etc.). Two do not recruit in the high schools, one because 
it was not permitted. All projects except one appeared to be responsive to walk-ins and referrals 
(2.53). What is not noted by RTI is whether the referring institutions or high schools gave the 
project information about the client that might assist the project in selection or in providing 
services. 

It would be useful to know how effective the different approaches to recruiting were. 
If TV and radio were as effective as school visits for recruitment, then programs might want to 
reassess their activities. The next evaluation could provide information about the effecti veness of 
alternative recruitment approaches tried by various projects. This could provide valuable 
information to other projects that have only tried one approach. 

There are two stated criteria for selecting clients: financial need and exceptional 
potential for postsecondarv education . Both evaluations considered how projects dealt with these 
criteria. 

While clients are supposed to be selected based on "financial need," RTI noted that 
several projects ignored this criterion on the assumption that 'Virtually all prospective clients were 
presumed to be in need and thus within the suggested guidelines. . . Each of these five projects was 
quite homogeneous in the area and population it served" (2.32). While this decision seems 
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reasonable, and it saves the program time (and cost) in dealing with clients, there is value in 
testing it given the central importance of this criterion in justifying and defining the program. 

RTTs assessment using projects' records concluded that for "those very few instances 
when both a family income and an individual's income were documented for a client, it was found 
generally that the combined incomes were low and/or the size of the family large" (6.17). This is 
somewhat reassuring, although the number of cases with complete information was too small to 
give much confidence. 

Unfortunately, RTI did not present tables for family income by family size. This 
would have been helpful for interpreting the data in their Table 6.15. Program guidelines said that 
H a non-farm family of 11 members could be eligible for Talent Search with an income of $10,400, 
with an added $700 allotted for each additional family member." (6.20). Since the definition of 
"low-income" depends on family size, we can only evaluate the appropriateness of clients with 
family income of $10,500 and over if we know their family size. Perhaps sufficient data were not 
available, but the absence of family-size information prevents these tables on family income from 
being clearly interpreted. The next evaluation must examine family income in conjunction with 
family size. 

In addition, the evaluation could test the project staffs' assumption that it is not 
necessary for staff to screen c li ents on income, since nearly all would be eligible . This is an 
example where an evaluation study can reaffirm a project's decision and help the project to keep 
its costs down. It might be possible for the evaluation to gather the financial aid information 
submitted to colleges. This might have been possible when the RTI survey collected information 
about enrollment from the colleges (in the survey RTI reported in Chapter 7). If this information 
supports RTFs tentative findings, then that could reassure program administrators that projects 
are serving the right participants. 

If so, the evaluation might allow projects to relax their information gathering. They 
might only be required to collect this information for a random sample of clients or for those 
clients seen in only two months a year. This will be discussed at greater length in the section on 
records. 
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It shouH be noted that the very aim of Talent Search tends to discourage youth from 
educated families from seeking to participate. Talent Search provides information that college- 
educated parents would already know. Moreover, middle-income parents with less education may 
have friends who could help their youth with this information, or they may send their children to 
schools that provide this information. Unlike other kinds of government programs which provide 
services that could benefit middle class participants, the services provided by Talent Search are of 
less value to ir'ddie class participants because they have other sources for this information. 

The second criterion for selection is "exceptional potential for postsecondary 
education." While clients are supposed to be selected based on "exceptional potential for 
postsecondary education," and many projects collected grades and test scores for such a purpose, 
RTI noted that only two projects used these criteria consistently in the selection of participants. 
Projects more often used referrals from high school counselors as their indicator of "potential." 

The next evaluation should investigate the advantages and disad vantages of high 
school counselors' referrals . Since high school counselors have access to students' whole records, 
they are in a much better position to assess potential than the Talent Search staff. However, RTI 
notes that several projects complained that counselors tended to either "cream" students too 
selectively or to refer "problem" students. The evaluation should investigate how pro jects dealt 
with inappropriate counselor referrals and if thev found wavs to get high scho ol counselors to offer 
more appropriate referrals. Improved counselor referrals would seem to be worth some effort in 
developing, since counselors could assist in assessing client strength and capabilities, attributes that 
most projects lacked good ways to assess quickly. 

However, RTT concluded that "it seems fair to say that overall, only a modest amount 
of attention was given to identifying "exceptional potential" for postsecondary education among the 
projects visited" (2.34). Several project directors felt that they could "find a place for any student 
who's interested" and others said that "all high school graduates (and dropouts and GED students) 
in the area deserved a chance and were deemed to have potential for some placement." 

The measurement of "potential" is difficult, and it would be a terrible misallocation of 
time and resources to devote much time to assessing potential in detail. Indeed, the directors' 
value judgments that all youth have potential may be warranted. But a related dimension, 
"preparation." is undoubtedly important to assess. 
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Franklin (p. 14) notes that poor preparation creates problems for assessment since 
"there is no way to distinguish between clients who are 'college-ready' and those who are 
unprepared to enroll in college because of work, family responsibilities, or inadequate academic 
background." While Franklin is mostly referring to the Educational Opportunity Center (EOC) 
projects in this quote, the same statement applies to Talent Search. 

However, this is where the evaluation can provide particularly useful information to 
directors. By examining the outcomes for various kinds of participants, the evaluation can identify 
which kinds of participants do not gain benefits from the program (or gain partial benefits, e.g., 
enrolling but not completing postsecondary education). 

Of course, unsuccessful outcomes can arise for a variety of reasons, not just lack of 
preparation. However, if the evaluation identifies some kinds of clients who do not ^ain benefits 
from the program because they are unprepared in some way, then that information can help the 
projects in selecting and advising clients. For instance, poor writing skills may only indicate poor 
previous instruction, but they can be a terrible barrier to success at four-year colleges, unless these 
colleges have remedial writing programs. Assessing these poor skills would influence the advice 
given to a student about which college to apply to. 

This is not to advocate a hard-nosed specific target rate of successful outcomes. 
Efforts to optimize success lead to "creaming, 11 which limits the number of clients served. 
However, information about the success rates of different types of clients can be useful to 
directors. If clients with poor writing skills do well in a few four-year colleges with special remedial 
writing courses, but do very poorly at other four-year colleges, then that will alter the advice staff 
give about which preparation youth should have before applying to most four-year colleges, or 
which college they should apply to. The evaluation should examine whether some k inds of 
information about clients can greatly help project staff to assess clients' p reparation and needs and 
thus improve the advice and services project staff give to clients. The emphasis should be less on 
screening people out of the program based on assessment of potential or preparation, as on giving 
them advice and services tailored to their particular preparation and needs. 
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Both previous evaluations supported the practice of considering clients as those who 
were contacted twice in individual contact. The evaluation should investigate what standards 
programs are now using to indicate whether a person is a "client." It is possible that programs may 
even use two different levels of contact, and if that were consistently defined, then it could be a 
useful way to evaluate the programs' effects. RTI has a good discussion of the meaning and 
measurement of "client" (2.38-44, 54-55) and the evaluation should respond to RTFs suggestions. 

RTI notes that all projects provided the following services: 

1. Information on educational opportunities and job training; 

2. Assistance with the mechanics of applications and admissions; 

3. Assistance with obtaining financial aid; and 

4. Arrangements for other types of assistance through referral. 

Regarding assistance with applications, "18 projects reported that they worked directly 
with clients in completing applications and often sent applications to institutions and financial aid 
sources-while in two projects, as a matter of policy, participants were expected to prepare and 
submit their own applications without direct assistance" (2.49). The next evaluation should 
examine how many individuals failed to get their applications submitted in t hose two projects. It 
should also evaluate the retention rates of these students compared with the rates for students who 
received program help in completing applications. Given the complexity of many of these forms, 
this hand-holding approach may be very appropriate, getting students over an unnecessarily high 
obstacle. However, one could also view some of this help as preventing individual responsibility, as 
the two programs assumed, and students who require too much help may not be prepared for 
postsecondary education. Both stories seem plausible, and the evaluation can help resolve which is 
true, or whether both are somewhat true. The answer to this question could be useful to projects 
in deciding how to allocate their time to various services. 

RTI also obtained information on academic counselling in 14 projects. Two projects 
engaged in counselling to a high degree and with apparent sophistication, another six made 
sporadic attempts with some success, three reported minimal amounts, and three offered none. 
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Franklin suggested that by 1982 more projects may have increased their emphasis on academic 
counselling, and perhaps shifted to younger age groups. The next evaluation should exa mine vhat 
benefits came from offering academic counselling, and which kinds of st udents benefitted. 
Academic counselling was not required by program guidelines, so the emphasis here should not be 
on using the success of academic counselling as an evaluation of the project. Rather, this should 
be used to provide information to other projects so they can judge whether they want to offer such 
a service and whether it may or may not meet a need for their clients or potential participants. 

Follow-up Services 

RTI reported considerable range of effort on follow-up once a client was accepted, 
ranging from no follow-up through extensive follow-up. Follow-up included taking clients to aid in 
registration, finding non-enroliees and providing ongoing assistance over several years of 
postsecondary schooling and even graduate school application. 

Obviously, such efforts are time-consuming and expensive, and it is reasonable for 
some projects to decide that these efforts are beyond their capabilities. Yet it is possible that some 
projects may have found ways to do these activities more economically. For instance, if a 
substantial number of clients are admitted to a single institution, a single staff member could help 
them all with registration. If a project has good relations with a postsecondary institution, then 
that institution might routinely notify the project about clients who fail to register, so the project 
can more easily assist those individuals. If cost is the main factor limiting these activities, projects 
might decide to provide these activities if they could see more cost-effective ways of doing them. 
The next evaluation should examine how various projects m anaged to provide these services and 
whether thev found ways to do them in an economically feasibl e manner. 

Project Staff 

The RTI study found that project directors and staff had strong backgrounds. For 
instance, 86 percent of project directors and 50 percent of project staff had a degree requiring at 
least four years of postsecondary education (3.17: 4.5). 
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Yet 33 percent of project directors had directed their projects for less than one year, 
and 66 percent for less than three years (3.17). Similarly, 40 percent of project staff had been in 
Talent Search projects for less than a year, and only 29 percent had been in Talent Search for 
more than two years (4.8). 

RTI noted that "Some concern was expressed about staff turnover, and in some 
locations it was attributed to a lack of job security in year-to-year funding situations and to lack of 
opportunity for upward mobility within [a] ... project. 11 The next evaluation sh ould consider what 
costs in terms of effectiveness arise from this lack of continuity . For a program that relies so 
heavily on community and institutional contacts and on highly specific information, the costs of 
discontinuity of staff and leadership may be enormous. Even if some uncertainty cannot be 
eliminated, there may be key positions which could be maintained with some security in order to 
minimize disruptions to the programs' operation. 



Project Records 

RTI has a good description of the deficiencies in recordkeeping (2.59-64). The next 
evaluation will inevitably revisit this issue in the course of its work. Recordkeeping is boring and 
often seen as irrelevant to serving clients, but it can be crucial for assessing and improving projects. 

RTI articulates a clear rationale for why programs might see recordkeeping as an 
irrelevant and wasted effort. Project staff "may have learned that systematic record-keeping on 
these large numbers [of clients] has proved to be of little use, since many may not continue with 
Talent Search after one or two contacts" (2.71). In essence, for a Talent Search project that spends 
only 5-20 minutes in individual contact with each participant, devoting another 5-20 minutes in 
record-keeping can reduce in half the project s effectiveness. Moreover, as noted, project directors 
and staff further viewed some of the central information requested (income and "potential") as 
unnecessary for various plausible reasons, so this difficult-to-get and somewhat embarrassing 
information was not always sought. 

All programs must face the difficult tradeoffs between time spent on services and time 
spent on recordkeeping, and this program may face a particularly severe tradeoff because it spends 
so little time with each client. Yet clear policies must be fashioned to confront these issues and to 
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balance the competing argument. Presumably, the program could stipulate some minimum 
amount of project recordkeeping which is crucial to assessing the projects. This means that the 
program must reach some clear goals that are central to assessing the projects. 

The next evaluation will discover whatever recordkeeping practices emerged after the 
RTI report, and it will be limited by these practices. Franklin's later report is not specific about 
how recordkeeping has changed since 1974, although he notes tha: recordkeeping was still not 
good. So it is not clear what records will be available for the next evaluation study. 

The RTI report was written for program administrators. It did not show project 
directors how records can be helpful to them. Future evaluators should make it clear to projects 
what value these records can have for individual projects in understanding their accomplishments 
and in assessing how their projects can be more effective. 

Good records can help program directors assess the outcomes of their various services 
for different kinds of clients. Records can help them assess how well their procedures are operating, 
and whether they are operating as they wish. Analyses of records can help projects decide what services 
to stress, which to improve, and which to decrease or drop. If project administrators are persuaded 
by such considerations, then they will be more likely to engage enthusiastically in recordkeeping. If 
not, then they will only begrudgingly do what the program mandates, and they will complain about 
"pressures" for recordkeeping from the regional office (3.59). 

The national program must decide which records it particularly wants to stress. The 
next evaluation can help. The RTI report stressed the importance of getting good information 
about outcomes, particularly "application for and acceptance in a postsecondary institution" and 
beginning "studies in a postsecondary institution 5 since "these two activity states reflect the major 
objective of Talent Search" (6.12). Moreover I would argue, these data are crucial for Talent 
Search projects to know in order for them to know how they are doing and to assess how to do 
better. 

Information on family income is also important. It is fundamental to the purpose of 
the legislation that set up the program. Although the records were not very complete on this 
matter, as on many others, RTFs assessment concluded that in "those very few instances when 



ERIC us l!8 



both a family income and an individual income were documented for a client, it was found 
generally that the combined incomes were low and/or the size of the family large" (6.17). 

Given the importance of this information for the mission of the program, the next 
evaluation must investigate family income . Moreover, in order for family income to be 
interpreted, analyses must also consider family size, since the income guidelines use different 
income limits for different size families (6.20). 

However, an evaluation can be used to test whether projects should collect complete 
information on family income. Even if a program occasionally made a mistake by collecting 
minimal information, and 5 percent of clients were not low income, it still might not be worthwhile 
to continue full collection of income data. The savings to the project in time spent screening 
clients could possibly be considerable. After alL if reducing screening and information gathering 
time allowed 25 percent more clients to be served, and 95 percent of those new clients were low- 
income, then there is an enormous net gain in the number of deserving clients served, even if 5 
percent of clients are over the income limits. 

Therefore, the next evaluation can test the adequacy of the present data that projects 
collect. The next evaluation can ascertain how often Talent Search projects mistakenly help youth 
over the income limits. It should also ascertain how much time it takes to get adequate information 
to reduce these mistakes. Consequently, it can infer what level of informatio n should be collected. 
Given the short length of time devoted to services to each client, and the time and difficulty 
required to get income information from clients, there are genuine tradeoffs involved. Moreover, 
given the fact that the very aim of Talent Search tends to discourage youth from educated families 
from participating because middle-class clients have other sources for this information, excessive 
concern about collecting income information may indeed be unnecessary. 

One compromise might be for complete record-keeping to be done only for a random 
sample of clients. Alternatively, staff might be required to keep complete records for only two 
months a year, and the other months would only require much reduced recordkeeping. 

Similarly, the evaluation can test the effectiveness of the projects' assumptio ns about 
youths potential ." If the evaluation finds that youth do manage to attend postsecondary 
institutions, and particularly that they are able to complete programs there, then there is little 



reason to question projects' procedures for assessing "potential" or to criticize projects for making 
little effort on this issue. However, if the evaluation finds less successful outcomes, then one 
possibility is that the projects are not screening students sufficiently well. 

Of course, this brings us to the issue of student outcomes, and it indicates how 
important that component of the next evaluation will be. 



Student Outcomes 



Student outcomes must be an important component of the next evaluation. RTI 
devoted considerable effort to this issue. Unfortunately, it belatedly discovered that it had 
overlooked some details, which made its data incomplete and hard to interpret. The next 
evaluation must be certain to avoid those mistakes. RTI reported four kinds of limitations of their 
data. I summarize these points (from pages 73-7.4) and comment on their lessons for the next 
evaluation: 



1. Programs used vague and inconsistent definitions of clients. Some included 
more than one year's clients, some included clients with only one contact 
(instead of the specified requirement of two or more), some counted only new 
freshmen, while others included transfers, reentering dropouts, and students 
returning for their second, third, or fourth years of postsecondary education. 
The next evaluation must specify the definition of clients more clearly. It is 
difficult to compare projects effectiveness if they have noncomparable client 
mixes. 

2. No attempt was made to determine if the client had enrolled in a different 
institution from the one reported by directors. This is difficult to study unless 
the evaluation or project does some follow-up of clients. It may be worthwhile 
to do a follow-up study of clients for a random sample or in a few projects. 

3. The evaluation did not follow-up drop-outs to see if they returned to school. 
This is expensive to study systematically, but like #2, can be done in follow-up 
surveys of students, especially on subsamples. 

4. Academic performance data (GPA, etc.) cover different time periods and 
numbers of credit hours. Of course, grades are relative and hard to compare 
across institutions, so it is not certain that more effort and expense should 
devoted to improving this data. The important fact in this information is 
whether the client is making passing grades. Therefore, the future evaluation 
should trv to obtain the postsecondary institutions* criteria for passing, and the 
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percentage of courses each student is passing . This would be the most useful 
information from such data. 

The first point is the most important. RTI discovered it too late to change their 
procedures, but the next evaluation can certainly take steps to avoid this problem. RTI suggested 
that the evaluation should specify that projects should include only clients receiving two or more 
contacts during the past 12 months and who were seeking to be beginning freshm en in the fall. 
This was recommended based on program guidelines at the time. Some may consider a lesser 
amount of contact to be an adequate service, which is also worth evaluating. That is an issue for 
the program staff to decide. 

However, the key point is that an evaluation requires a consistent definition. If we 
compare outcomes from clients receiving different numbers of contacts, we risk getting a mistaken 
impression. 

RTFs reported results are very clear and well presented generally, but I am 
disappointed that they stopped where they did. While it is useful to know the number of clients for 
whom "enrollment status is undetermined," it would also be useful to exclude the "undetermined" 
group in assessing the percentage with successful outcomes. RTI did not do this. RTI notes that 
enrollment "status was undetermined if: a) the institution in which he was reported to have 
enrolled did not return the verification form or b) the institution responded but for some reason 
did not provide information on the client" (7.16). If we treat these as missing cases, and remove 
them from the total in each comparison, then we can get a better indication of the success rate of 
each group. 

It is hard to know what biases arise from excluding registrar nonresponse. We would 
expect most registrars to indicate if a student is not on their records. However, some registrars 
may fail to respond when they cannot find a name in their records, so both categories of 
nonresponse may be slightly correlated with nonatten dance. Deleting these cases may slightly 
inflate the success rate, but this is unlikely to be a large bias. While we must bear that in mind, 
omitting nonresponses allows us to calculate a more useful success rate. 

For instance, the RTI report's Table 7.2 shows that 72 percent of freshmen were 
enrolled in the fall, 11 percent were not, and 17 percent were undetermined. If we drop 
undetermined from the total, so the total is only 83 percent of the freshman group, then the 
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enrollment rate is 72/83 = 87 percent. The mixed group (including freshmen, returning 
postsecondary dropouts, and transfer students) has a rate of 66/90 = 73 percent, and the 
insufficient listing group has an enrollment rate of 78/97= 80 percent (see revised Table 7.2, at the 
end of this paper). Granting that these somewhat exaggerate the enrollment rate (if we assume 
that the "undetermined" includes somewhat more nonenrollees), they are probably better 
estimates of the enrollment rate than the numbers given in the original RTI table. Moreover, 
these numbers increase the enrollment rates substantially for the freshman and mixed groups, and 
they bring the freshman group to a higher rate than the insufficient listing group. 

Substantively more interesting are the adjustments applied to Table 7.6, which reports 
the initial enrollment status by type of institution. In a new version of Table 7.6, we report the 
original enrollment rates and the adjusted rates (removing undetermined from the totals) (cf. 
revised Table 7.6, at the end of this paper). 

By removing "undetermined" from the totals, the enrollment rates increase. What is 
interesting is that the initial apparent difference between two-year and four-year public schools 
disappears after the adjustment. The difference between the two-year and four-year private 
schools increases somewhat, but this is not significant. The other schools (vocational, technical, 
business or other proprietary schools) continue to have a low rate, but it is not as low as the 
unadjusted number seemed to indicate. 

Similarly, I made such adjustments to Table 7.8, on the retention rate (literally, the 
enrollment status in spring of 1974 by type of institution in which client enrolled), (cf. revised 
Table 7.8, at end). Here, the apparent difference between private and public two-year schools 
disappears after adjustment, while the apparent similarity between private and public four-year 
schools disappears after adjustment. After adjustment, four-year private schools have a somewhat 
higher retention rate than comparable public schools. 

The same adjustment was applied to Table 7.10, on the percentage of those leaving 
school who dropped out, by type of institution (cf. revised Table 7.10, at the end of this paper). 
Again, the apparent difference between private and public two-year schools disappe: ^ after 
adjustment, and the percentage choosing to drop out from four-year public schools is much greater 
than the original data implied. The low "self-chosen dropout" percentage for four-year private 
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institutions reflects 25 percent (adjusted) who were dropped by the institution, a much greater rate 
than other groups, which ranged from 3 to 8 percent. 

This is an area of some concern, worthy of further exploration by the future 
evaluation. If 25 percent of those leaving are being dropped by the institution, then Talent Search 
projects should be aware of the reason, and this might affect which students they direct to such 
schools. It should not be hard to ask the school its reason for dropping the student. 

Incidentally, this 25 percent is only 87 individuals, or just 3 percent of clients choosing 
to attend such schools. As noted, the overall retention rate was greater in private four-year 
schools than in public four-year schools. 

Finally, we look at grade point averages by type of institution. Of course, schools 
vary in their grading scales. The most important aspect of grades is whether students are getting 
acceptable grades (which I'll define as 2.0 or higher), and whether they are getting failing grades 
(which HI define as less than 1.0). Of course, the better way to measure these categories would be 
to ask the institution what its criteria for "acceptable grades" and "passing grades" are and to get a 
count of the number of grades that are above and below these criteria. 

Again, we copy the number reported by RTI, and then adjust those numbers after 
deleting individuals whose grades were not reported. Here we expect no bias from excluding those 
individuals from the basis of calculation, since non-report is often due to institutional policies 
(pass/fail grading systems, or policies against reporting grades without student permission) (cf. 
revised Table 7.12, at end). 

While the original data imply that students at four-year public schools are less likely to 
get acceptable grades than 4-year private schools, the two are virtually identical after adjustment. 
They are even somewhat less likely to get failing grades, a difference that remains after 
adjustment. While most groups look identical in their original failure rates (at about 10 percent), 
after adjustment, the two-year public and four-year private schools have the higher rates of failing 
grades. Interesting, these are the two types of schools at the lower and upper ends on selectivity, so 
the reasons for their higher rates of failure may be quite different. Of course, the highest failure 
rate is among the "other" schools (vocational, technical, business or other proprietary schools), 
which again points to some concern about these schools. Recall that these adjustments definitely 



improve our estimates, since the institutions that don't report grades are unlikely to indicate any 
bias about kinds of individuals. 

These are important findings. Despite all the problems with these data, they point to 
potential differences in the success rates at different kinds of institutions. The evaluation should 
notify projects about these outcomes, so thev can take account of them in their advising. 

Besides performing these adjusted analyses, the future evaluation should try to 
minimize nonresponse. to clarify the reasons for institutions not respondin g about an individual 
(by giving instructions about how to handie individuals not listed), and to ascerta in the reasons for 
nonresponse . By taking such steps, better estimates can be made at no additional cost. 

Of course, we should also not ignore one of the simpler and more important findings, 
that approximately 70 percent of participants attending postsfecondary schools went to four-year 
colleges. This is a strong effect, especially when contrasted with the general tendency of minorities 
and low-income populations mostly to attend two-year colleges or other schools. 

Local Variations 

Evaluations usually try to gauge the effectiveness of entire programs and then 
compare the effectiveness of various local ones. But to be most helpful to local projects, 
evaluations must be sensitive to local variations in the requirements and particular advantages of 
local situations. 

For instance, some postsecondary schools offer strong remedial programs while 
others do not. Similarly, some postsecondary schools offer strong job placement rates while others 
do not. The ultimate effectiveness of a Talent Search project depends on how well its placements 
are appropriate to students' particular needs (e.g., for remedial writing) and to job availabilities in 
the area. This cannot be assessed nationally; it must consider particular local circumstances. 

To the extent possible, the evaluation should analyze way s Talent Search projects 
gather information ah ""* locaj Eostsecondap; schools (and prog rams within schools) and l ocaljub 
availa bilities, and the wavs projects use thi s local information. 
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In addition, the evaluation should be sensitive to differential effects of different types 
of placements . While the evaluation cannot assess the effectiveness of various local postsecondary 
schools, some effort should be devoted to locating institutional research results in some of the 
major postsecondary schools referred by the Talent Search projects. In addition, if it is feasible to 
do a long-term (five or more years) follow-up of Talent Search participants, this survey would 
permit analyses of the effectiveness of various placements. (It is not certain that a five-year follow- 
up is feasible, given the difficulty of locating people after that long a period.) 

The evaluation should also be sensitive to differential effects of different types of 
projects . In particular, projects which provide academic tutoring and assistance may have greater 
benefits on reducing drop-outs and improving postsecondary completion. It is also possible that 
Talent Search projects are too low in intensity to have much effect on academic skills. The 
evaluation should examine whether Talent Search projects which pro vide academic services can 
effectively improve youths' academic achievement. On the one hand, other programs may provide 
referrals to agencies which can give more extensive academic services. The benefits of using 
referrals to improve academic services should be examined. 

Perceived Problems and Accomplishments 

Project directors' and staffs' perceptions of problems can be useful to program design 
and also to program administrators and staff. It helps program staff to understand and evaluate 
their own experiences, and feel better about their successes and feel less bad about their failures. 

The early RTI report noted problems of insufficient staff personnel (87 percent), 
difficulty obtaining support from parents (59 percent), inadequate facilities (54 percent), 
difficulties with student recruitment (46 percent), and problems obtaining cooperation from 
students (47 percent). The future evaluation should eva luate project directors' and staffs' 
perceptions of problems, and it might extend this line of inquiry to investigate how project staff 
dealt with these p roblems, and which approaches seemed most successful. 

Many projects noted the difficulties of staff training. One way to reduce this problem 
is to reduce staff turnover, as noted above. Another way is to get technical assistance from the 
U.S. Department of Education regional office, and the evaluation should monitor this. 



RTI reported that project directors and staff perceived many significant 
accomplishments. Without repeating the full list of accomplishments (cf. 2.67-2.68), it is 
particularly striking that "there was a general sense that adequate financial aid packages were 
obtained for the majority of the participants served/ that projects influenced postsecondary 
institutions to improve campus counselling, financial aid, admissions policies, etc., and that 
projects created "good working relationships with the community, particularly the ethnic 
community." (2.67). RTI also noted several exemplary special efforts of projects, including 
administrative arrangements to improve ease of getting financial aid and test scores to participants 
(2.69). Of particular interest, given the sometimes mistrustful relationships with high schools, one 
project hired its director and counselor from the staff of the local high school, on a leave of 
absence. This arrangement not only gives the project experienced staff, but it also sows the seeds 
of cooperation between the project and the schools in future years. The current evaluation should 
also investigate directors' and staffs' perceptions of success and of the various exemplary practices 
which thev felt improved their operations . 

Interactions with High Schools 

RTI had some information on this (cf. pp. 2.56-57), and Franklin mentioned it briefly. 
But this is a crucial potential asset which deserves greater attention than previous studies devoted. 

Talent Search exists because it is assumed that high schools are not doing an adequate 
job of helping disadvantaged and low-income youths get higher education. Therefore, the 
existence of Talent Search may be interpreted as a school failure, and thus threatening to the 
school. Projects differ in the cooperation they receive from high schools. Some difficulties may be 
inevitable, since some high schools may see Talent Search doing activities that the school is 
supposed to be doing. How do programs negotiate their relationships with schools? Are there 
ways they allay school staff defensiveness and offer their services to assist schools? Are there ways 
that Talent Search programs provide information to help improve schools' capabilities for college 
counselling? 

Another possible risk is also evident. The RTT evaluation reported the "tendency of 
high schools to do less counseling once a Talent Search project has begun in the area, with a 
consequent dependence on Talent Search to do the high schools work in academic counseling, 




provision of information on colleges and financial aid, and provision of assistance to 
students. H (2.66). This carries serious risks of hurting services to the very youth that Talent Search 
seeks to help. Of course, if the schools were doing nothing, then little is lost. But otherwise, 
Talent Search may contribute to a decline in schools' assistance. To the extent that schools have 
good contacts with some post-secondary institutions that Talent Search does not, or that schools 
provide some services that Talent Search cannot (e.g., few Talent Search projects provided good 
academic counselling), then schools' reduced services are a potentially serious concern. 

The evaluation should investigate in what circumstances high school s reduced 
services, and how various projects dealt with high schools to prevent this . It seems likely that 
projects can work out a division of services to give both counselors and projects distinctive roles. 
The evaluation should examine how this has been done. 

As Franklin noted, there is a constructive role that Talent Search can play in 
supplementing the meager capabilities of schools (Franklin, p. 15). Indeed, Franklin notes that 
"there are practically no college guidance services in the 10 high schools in the Talent Search area" 
of San Antonio, (p. 12). Obviously, in San Antonio, Franklin sees no risk of reducing the high 
schools' services, which are practically nonexistent. 

The key issue is how Talent Search projects define and differentiate their services 
from high schools and how they reach agreements with high schools. The evaluation should 
investigate how some Talent Search projects successfully negotiated the re spective roles for 
themselves and for the school counselors so that students received increas ed services, not just a 
shift of responsibilities . RTI alludes to these issues, but if the current evaluation were more 
specific, it could help projects assess how projects handle these dilemmas. 

Relationships to Postsecondary Institutions and Other Institutions 

Talent Search projects' relationships to postsecondary institutions are less 
threatening, but programs seem to differ in how closely they can work. Here there should be 
some similarities of purpose, since many postsecondary institutions, having seen declines in 
minority enrollment over the past decade, are seeking to increase minority enrollment. Research 



124 



should investigate how Talent Search programs have established relationships with postsecondary 
institutions, and the ways they have found to strengthen their cooperation. 

RTI reported that "many project directors had developed favorable relations (in terms 
of acceptance, financial aid packages, or both)" with some postsecondary institutions (2.50). 
However, some difficulties in working with some institutions arose, especially in allowing fee 
waivers (2.67). The current evaluation should investigate Talent Search projects' relationships 
with postsecondary institutions and what procedures led to successful arrangements . 

On the other hand, some problems were noted of "competition for clients by other 
agencies (community, armed forces, industry) serving or seeking the same population" (2.67). 
This is not necessarily a problem - it indicates that youths were getting a multitude of 
opportunities- no: a common situation for most low-income, disadvantaged youth. However, the 
current evaluation should investigate the nature of this competition among different ag encies, and 
how the respective organizations deal with it . For instance, lx employers are drawing talented 
youth away from higher education and better long-term career options and into jobs that offer 
poor future prospects, then Talent Search may need to address this dilemma in its counseling. 

As noted in Section 1, Talent Search is designed to offer mostly information, advice, 
and counsel over a short period of time. Consequently, its effectiveness may increase if it is able to 
enlist the help of others - church groups, parents, Big Brother and Big Sister programs, and other 
volunteer efforts. The evaluation should examine to what extent Talent Search projects are able to 
enlist the help of others. 



3. Which research methods would make sense to use in evaluating Talent Search given 

the nature of the program and the questions we might want to answer? 

The preceding section identified issues for study in the future evaluation. The RTI 
evaluation used a diversity of studies: they surveyed project directors, project staff, advisory board 
members, previous participants, postsecondary school registrars. RTI also did site visits to 20 
projects and analyzed projects* records. If funding were unlimited. I would call for repeating the 
whole process. However, if funds are limited, I would focus on certain kinds of information. 
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The key issues, in my opinion, are clients' outcomes. The success of the program 
ultimately depends on whether clients attend postsecondary schools, what kinds of schools they 
attend, whether they complete programs, and how they do in those programs. Therefore, I would 
place highest priority on studies of clients' outcomes. These could be studied by surveys of 
registrars at postsecondary schools served by the projects. 

However, since youth often transfer to different institutions from those they first 
attend, a client survey is probably necessary. This may present some difficulties. Low-income 
populations tend to be highly mobile and hard to locate, and this is more true for youth. However, 
if the program has collected sufficient information about participants (Social Security numbers and 
family address), then reasonable proportions may be found. 

Second in priority, I would stress analyses of projects' records. While these records 
were not good in the RTI survey, they may have improved. We don't know how good they are 
now. If they are not much better, they may possibly be supplemented by interviews with staff and 
by information gained from financial aid information from college registrars, although this may 
require student permission. 

Third in priority, I would endorse a survey of staff, directors and advisory boards. 
These are the easiest groups to study, and RTI obtained a high response rate from each. The RTI 
surveys showed that staff and directors were well qualified but highly mobile, and that may be 
useful to test again. Perhaps more useful would be to use interviews with project staff and 
directors about the procedures they use to do their work, and their suggestions for other projects. 
I discuss this at greater length in the next section. 

In addition, these surveys could examine the services projects get from regional 
offices. RTI reported that "while USOE (U.S. Office of Education, now the U.S. Department of 
Education) regional offices reportedly provide a wide range of services to most of the Talent 
Search projects, a substantial number of project directors that they did not receive technical 
assistance in program development, were not visited by a regional staff member, or, if visited did 
not receive a summary of the site visit; did not receive technical assistance in data collection, and 
did not have the opportunity to attend proposal writing workshops"(Vol. Ill, p. 5). The evaluation 
could examine whether the assistance from regional offices has improved. 
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Staff in some projects were concerned "over the continuous pressure from regional 
offices to serve large numbers of participants, with the implication that projects would be 
evaluated and funded in terms of the number of clients reported." (Vol m t p. 5). Obviously, there 
is great value to serving many individuals, but there may also be a trade-off between numbers and 
amount of help provided. While this program seems to be designed to reach many individuals, and 
that seems an appropriate goal, excessive emphasis on this could dilute program services to be too 
superficial to be very helpful at all. Holding large meetings to provide information can inflate the 
number of participants while possibly being ineffective at attaining the program's goals. The next 
evaluation could examine the views of project directors and staff on this question. 

The impact of the whole Talent Search program is difficult to assess since there is no 
control group. Most Talent Search programs seive all who seek help, so we cannot know what 
unserved people to compare with this self-selected group. RTI notes concern that this self- 
selection "tends to bypass potential participants who do not respond to "general" invitations M (Vol 
HI, p. 4). However, using self-selection as a selection criterion may be reasonable given that 
considerable motivation and self-initiative are needed in the college application process and in 
succeeding in college. 

I do not see anyway to construct a suitable "control group" to judge the overall effects 
of Talent Search. Yet comparisons within Talent Search can be very useful. Comparisons of the 
effects of different services and different types of projects can be particularly valuable. 



Useful Information for Projects 

As Fve noted throughout, many of the above analyses can provide useful information 
to project directors and staff. But there is also another kind cf information that evaluations can 
provide which may be seen as valuable by project directors and staff. Information about the 
methods that various project directors and staff have developed for doing their jobs is often useful 
to other project directors and staff. Information on the kinds of problems experienced in other 
projects can help staff and directors to understand their experiences and to not blame themselves 
when they fail with some cases. 
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Project directors and staff could benefit from answers to the following questions. 
What information is most useful to clients? What do low-income youth need to know? What 
information gives them motivation? What information tells them what steps to take? What 
misconceptions or problems of youth are hard to detect but may undermine their success? How do 
project staff reach "hard-to-reach" kids? What causes project failures, and how might they be 
saved? 

Another kind of information that can be useful to projects is general information that 
staff can use to present to clients. For instance, the field of college choice, financial aid, college 
drop-outs, etc. are complex, and extensive research and source books are available. Some kinds of 
information may have to be specially developed to serve these low-income populations. Having 
this information centrally compiled would offer efficiencies over having each local project compile 
its own information. If staff knew more of this information, they might do a better job of helping 
their participants. 

The evaluation can gather this information from various sources. Regional office staff 
may have some of this information. The staff at various projects may have relied upon different 
sources of information and they may have devised various approaches to counselling. The 
accumulated wisdom from the practices of various projects should be shared so other projects can 
benefit from it. 

Moreover, if the Talent Search program developed materials on college applications, 
financial aid, preparation to reduce drop-outs, and tailored this material to low-income 
populations, these materials could contribute to the effectiveness of urban high-school counselors 
in helping these youth. While some schools in low-income areas have no counselors, and some 
counselors in such schools lack the time or interest to help low-income youth attend college, many 
school counselors want to help these youth, but lack the information about how to help them. 
Obtaining financial aid may be a particularly obstacle in their view. If Talent Search can provide 
high school counselors with such information, that may improve the schools' ability to deliver such 
services to low-income youth. 

Consequently, as noted in the first section on general policy questions, the next 
evaluation should pay some attention to the shortcoming s, resources, and needs of the local high 
schools in the Talent Search locations. In addition, the evaluation should examine what prevents 
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schools from identifying youths with potential or what prevents them from helping these youths . 
Such an evaluation will help Talent Search projects to assess how better use can be made of the 
schools for helping these youth. 

Having the evaluation deal with some of these issues pushes evaluation into terrain 
that has not been examined, although some of the site visit material in the RTI report alluded to 
related information. However, this kind of evaluation may increase the evaluation's usefulness to 
projects and may increase the projects' commitment to the evaluation process. 



Conclusion 

RTI suggested that "Counselors have learned that in any event they must await the 
cues from the clients: they cannot force, they may only encourage. . . they may have evolved the 
most practical approach under the circumstances. Talent Search is more akin to a missionary 
effort than a defined program for carefully selected or committed participants. In this respect, it 
differs considerably from Upward Bound, for example, in that it is essentially unstructured and 
depends to a considerable degree upon participant responsiveness at the outset as well as in 
subsequent contacts" (2.71). 

This seems an apt way to characterize the program, and it provides a cautionary note 
for those doing the next evaluation. This is not the usual highly intensive program on a distinct set 
of individuals. It is much larger than the usual program and the target group is more diffuse. Yet 
the impact may still be very great if the program is effective at its mission. That is what makes the 
next evaluation both challenging and potentially very rewarding. 
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REVISED TABLE 12 Original and Adjusted Enrollment Rate by Group 



Original 
Adjusted 



Freshmen 
only 



Mixed 
group 



Insufficient 
Listing 



72 
87 



66 
73 



78 
80 



REVISED TABLE 7.6 Original and Adjusted Enrollment Rates by School 



2-year 
public 



2-year 
private 



4-year 
public 



4-year 
private 



other 



Original 
Adjusted 



76 
81 



85 
90 



68 
81 



84 
87 



38 
68 



TOTAL 



71 
82 



REVISED TABLE 7.8 Original and Adjusted Retention Rates by School 



2-year 
public 



2-year 
private 



4-year 
public 



4-year 
private 



other 



Original 
Adjusted 



62 
65 



65 
65 



76 
78 



77 
83 



53 
56 



TOTAL 



72 
75 



REVISED TABLE 7.10 Original and Adjusted Percentage Leaving School Who Dropped-Out by 
School 

2-year 2-year 4-year 4-year other TOTAL 

public private public private 



Original 
Adjusted 



72 
81 



82 
82 



68 
86 



51 
68 



78 
88 



68 
63 



REVISED TABLE 7.12 Original and Adjusted 

Percent Students with Acceptable Grades (GPA 2.0 or higher) by School 



2-year 
public 



2-year 
private 



4-year 
public 



4-year 
private 



other 



Original 51 74 
Adjusted 66 76 



39 
63 



50 
64 



41 

59 



TOTAL 



44 
64 



Percent Students with Failing Grades (GPA less than 1.0) by School 



Original 
Adjusted 



2-year 
public 



10 
13 



2-year 
private 



4-year 
public 



4-year 
private 



5 

8 



10 
13 



other 



10 
15 



TOTAL 



7 
10 
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MEASURING PROGRAM IMPACT: 
WHAT IMPACTS ARE IMPORTANT TO ASSESS, AND 
WHAT IMPACTS ARE POSSIBLE TO MEASURE? 
A PROPOSAL FOR RESEARCH 

William T. Trent 



L Introduction 

This paper proposes a multimethod research design for the assessment of the impact 
of Talent Search, a federally funded program originated in the Higher Education Act of 1965. The 
proposed research is the first of its kind targeting Talent Search and proposes a data collection and 
analysis strategy most suitable for identifying the impact (effects) of Talent Search services on its 
direct and indirect clients. The direct clients of Talent Search are those students who benefit from 
the information and counseling services provided by the funded programs. The indirect 
participants are the parents and families, school counsellors (grades 6 through 12), college 
admissions officers and college financial aid officers. In order to adequately determine the impact 
of Talent Search services, members of each of the above client categories must provide responses 
to key questions covering the range of services provided by Talent Search programs. 

The most general goal of the proposed research is to identify the impact of Talent 
Search services. To achieve this goal, the research should also: (a) establish a baseline of service 
indicators drawn from a rich description of Talent Search services; (b) develop and administer 
baseline measures of Talent Search services; and (c) identify those conditions that determine the 
comparability, or lack thereof, between Talent Search projects. 



2. Background 

Talent Search has its origins in the Higher Education Act of 1965, Title IV, Sections 
417A and 417B. Educational Opportunity Centers (EOCs) were established by the Education 
Amendments of 1972 and centers began operating in 1974. These centers also provide Talent 
Search services. Throughout this paper, when reference is made to Talent Search services, it is 
inclusive of EOCs. The purpose of Talent Search is to identify qualified youths with potential for 
postsecondary education, to encourage them to complete secondary school and to enroll in 
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postsecondary education programs, to publicize the availability of student financial aid, and to 
increase the number of secondary and postsecondary school dropouts who re-enter an education 
program. Services provided through Talent Search are designed to increase rates of high school 
graduation and improve college preparation and access for a disadvantaged and largely minority 
population. 

The targeted age group for services is comprised of disadvantaged individuals 
between the ages of 11 and 27, two-thirds of whom must be low-income and also potential first- 
generation college students. The services provided by Talent Search programs include: assistance 
in completing college applications and financial aid applications; preparation for admissions tests; 
academic, financial and personal counseling; tutoring; career exploration and aptitude assessment; 
and a variety of information services including high school course-taking options, types of colleges 
and universities and types of financial aid; and assistance with reentry to high school or college. 

Research to date has been largely descriptive of the services provided by Talent 
Search and, as a consequence, it has been difficult to determine the value of this program. What is 
certain is that the conditions that led to the legislation authorizing Talent Search and its 
companion programs have not lessened. It is rather the case that those conditions have become 
more exacerbated in many ways, especially in terms of the quality and amount of information that 
disadvantaged and minority youths have in formulating their educational choices. 

3, Conceptualization And Research Questions 

Assessing the impact of Talent Search begins first with the determination of the 
principal aims of Talent Search, and hence, the possible impacts. The proposed research 
conceptualizes the program as one which realizes its impact once students or potential students 
have (a) acquired an understanding of academic preparation necessary for postsecondary 
enrollment, (b) received information explaining the process leading to postsecondary enrollment 
and exposure to postsecondary options, (c) received information about the application process for 
admission to and obtaining financial aid for postsecondary education, and assistance in completing 
those applications, and (d) received counseling regarding the value of completing secondary 
schooling and pursuing postsecondary credentials, including the attendant career options. This 
conceptualization of the Talent Search program stops short of assuring enrollment, re-entry, or 
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completion of secondary or postsecondary levels of schooling. Such assurances, when Taient 
Search programs are held accountable for these outcomes, are probably unrealistic given the 
variety of factors that shape these outcomes and the expenditure level for Talent Search. The key 
phrase here is "held accountable". It is reasonable to expect the services of Talent Search to lead 
to college enrollment; however, Talent Search is a necessary service for its clients but not sufficient 
to determine enrollment at the postsecondary level. Other factors, including the quality of financial 
aid (amount, duration of aid, proportion gift versus loan), alternatives such as the military or 
pressures to work, in addition to the admissions requirements or college proximity, could 
independently or collectively prevent college entry. As a result, the proposed impacts, detailed 
below, but derived from this conceptualization, are viewed as important and measurable but stop 
short of requiring high schorl graduation or college entry as an accountability outcome for Talent 
Search. Talent Search programs also have relatively limited contact with their clients compared to 
Upward Bound or Special Services. One consequence of this program design is that the services 
are both more 'tailored' and more restricted. In effect, Talent Search provides rich, comprehensive 
advising, counseling, and assistance, but on a focused set of topics. This program design is what 
allows Talent Search to achieve its goals at very low per-client costs, relative to its sister programs. 

Finally, in light of the above conditions it is important to consider the program 
implications if actual college enrollment is treated as an accountability outcome. Such a criterion 
could alter service development and delivery decisions in ways that could reduce the overall 
number of clients served. For example, in my tenure as a project director, I might have felt 
compelled to direct a disproportionate amount of project services to those clients who showed 
"clear potential" for college entry and success at the expense of casting a broader net to capture 
those clients whose potential was less clear. In effect, project services would then go largely to 
those students who present more traditional college-bound student profiles in order for the project 
to appear more effective. This would not have the desired effect of "expanding the pool" of 
potential college students among low-income, largely minority students. A further complication is 
that it would also be difficult to establish any fixed standard or range of "high school graduates" or 
"numbers of clients enrolled in college" that would be fair /just across projects or even for a single 
project from year to year. Nonetheless, information on whether or not sti.dent respondents in the 
proposed study do graduate from high school and/or enroll in college should be collected. 

Drawing on the above conceptualization of Talent Search, the central research 
question for this study is: What services do clients of Talent Search programs report receiving, 
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and how do they assess the importance of those services for their educational careers? Stated as 
hypotheses to be tested: 

k Clients who report having received services from a Talent Search program are 
likely to report being informed about the academic preparation necessary for 
postsecondary enrollment; 

■ Clients who report having received services from a Talent Search program are 
likely to report having received information explaining the process leading to 
postsecondary enrollment and exposure to postsecondary options; 

■ Clients who report having received services from a Talent Search program are 
likely to report having received information about the application process for 
admission to and obtaining financial aid for postsecondary education and 
assistance in completing those applications; and 

■ Clients who report having received services from a Talent Search program are 
likely to report having received counseling regarding the value of completing 
secondary schooling and pursuing postsecondary credentials, including the 
attendant career options. 

These hypotheses do not require an assumption that each Talent Search project 
delivers these services in an identical format but rather that these are the categories of services 
that participants receive. Each hypothesis centers on the delivery of either counseling, advising, or 
direct assistance targeted to one or more of the legislated Talent Search service provisions. Since 
all Talent Search projects must comply with the authorizing legislation, these hypotheses seem 
sufficiently broad to encompass a variety of service delivery formats. 

The research question stated above also entails using indirect participants as a source 
of important data in assessing the impact of Talent Search services. The indirect participants of 
Talent Search services are parents, school counselors, and college admissions and financial aid 
officers. Questions exploring these hypotheses are appropriate for parents of pre-high school and 
high school participants, including dropouts. Parents of participants enrolled at the postsecondary 
level are also included, as they often return for assistance with annually submitted financial aid 
forms. 

Counselors at the pre-high and high school level are direct participants in some 
instances, inasmuch as Talent Search programs often provide direct assistance to them in the 
provision of applications, forms, and other college related materials that may be difficult to access 
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or unavailable through local system channels. As a former Talent Search director, I recall 
personally delivering College Scholarship Service Family Aid Forms (FAF) to high school 
counselors at the beginning of the school year and during the fall school term when they were 
unable to secure such forms through school channels in an expeditious manner. Most often, 
however, counselors are indirect participants, with a primary concern for accessing available 
educational opportunities for their students. Talent Search services are of special value to those 
counselors who work in schools/districts that enroll large numbers of the target population 
because of the usually large student-counselor ratios and the broad range of counseling needs such 
participants present. As a result, Talent Search programs augment/supplement the school 
counseling services. For example, Talent Search programs develop and conduct school assemblies 
for students in conjunction with the counselors for the junior and senior classes. When serving as a 
project director, I conducted such assemblies in the Washington, D.C. high schools on a regular 
basis, the Talent Search project becoming in effect a planned part of the services available to the 
students in those schools. This places school counselors in a unique role for experiencing the 
services of Talent Search programs and make them ideal candidates for being respondents in 
research assessing the impact of Talent Search. 

A third category of indirect participants are college admissions and financial aid 
officers. Many Talent Search programs operate with direct access to admissions and financial aid 
officials at the postsecondary level, especially within, but not limited to, their local area. Moreover 
several Talent Search programs hold annual College Fairs attended by a wide range of 
postsecondary institutions, often providing at least a regional representation, and sometimes 
approaching national representation. Many of the admissions counselors/recruiters thus look to 
Talent Search programs as an important resource for identifying talented disadvantaged and/or 
minority youths. Two Talent Search projects with a long history of "College Fair" programs 
attended by college representatives are the Educational Opportunity Center in Washington, D. C. 
(Project OPEN) and the Chicago, Illinois project offered through the Ada S. McKinley House. 
Both projects have a sustained history of work with college admissions officers and recruiters. 
Moreover, as a former director of a college-based educational opportunities program (the EOP at 
George Washington University), I can personally attest to having recruited students through 
Project OPEN and to having referred participants to those services. Like secondary school 
counselors, then, college admissions and financial aid officials have had a unique opportunity to 
experience both the program services and the clients of the programs and are therefore an 
important source of data about the impact of Talent Search serv ices. 
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The research question which engages counselors and postsecondary officials is: What 
is (are) the effect(s) of Talent Search services on your students and institution? This question can 
be explored in ways that will provide insights into the impact of Talent Search services from the 
purview of educators who can report on their perceptions of the effects of Talent Search on its 
clients as well as their perceptions of the role of Talent Search in enabling their institutions to 
serve the target population. Hypotheses appropriate for school counselors and college officials 
might include: 

■ School counselors who report receiving services from Talent Search projects 
are likely to report having been assisted in developing informational programs 
for their students regarding the benefits of high school graduation; 

■ School counselors who report receiving services from Talent Search projects 
are likely to report having been assisted in developing informational programs 
for their students leading to enrollment at the postsecondary level; 

■ School counselors who report receiving services from Talent Search projects 
are likely to report having been assisted in securing materials (financial 
aid/admissions applications, etc) for their students leading to enrollment at the 
postsecondary level; 

■ School counselors who report receiving services from Talent Search projects 
are likely to report having their students receive direct assistance in completing 
materials (financial aid/admissions applications, etc) required for enrollment at 
the postsecondary level; 

■ College officials who report receiving services from Talent Search projects are 
likely to report having received direct assistance in gaining access to increased 
numbers of low-income and minority students who are potentkily eligible for 
enrollment at the postsecondary level; and 

■ College officials who report receiving services from Talent Search projects are 
likely to report having received assistance through the referral of students and 
parents of students, who are enrolled or potentially eligible for enrollment at 
the postsecondary level to the project for assistance with the completion of 
applications and forms required for college entry/financial aid; 

In summarizing, the critical areas of Talent Search impact should include the 

following: 
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Impact on Students/Potential Students 



Grades 8 - 10 



Clients should report receiving: 

involvement in activities (Talent Search Clubs, assemblies, and workshops) that 
impart information on the importance of staying in school and graduating; 

involvement in activities that impart information on postsecondary education: 

involvement in activities that deliver information on course taking options that 
prepare students for entry to the postsecondary level; 

exposure to postsecondary institutions and representatives of postsecondary 
institutions; and 

involvement in activities that inform students about practices that make for 
being a better student. 

Grades 11-12 



Clients should report receiving: 

counseling and advising about the reentry process (dropouts); 

assistance in the re-entry process (dropouts); 

direct access to representatives of postsecondary institutions; 

counseling on test preparation and application procedures for taking the test, 
especially the preliminary SAT and the SAT; 

counseling and advising on courses that must be taken in order to satisfy 
postsecondary entry requirements. This is especially necessary for students with 
the potential to enter four-year colleges and universities; 

counseling, advising and assistance in the application process for admission to 
postsecondary institutions; 

counseling, advising and assistance in the application process for financial aid 
at the postsecondary level; and 

involvement in activities that inform students about practices conducive to 
being a better student. 
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Impact on Students and Potential Students at the Postsecondary Level 

Clients should report receiving: 

counseling and advising about the reentry process (dropouts); 
assistance in the reentry process (dropouts); 

counseling on test preparation and application procedures for taking the test, 
especially the preliminary SAT and the SAT; 

counseling and advising on courses that must be taken in order to satisfy 
postsecondary entry requirements. This is especially necessary for student with 
the potential to enter four-year colleges and universities; 

counseling, advising and assistance in the application process for admission to 
postsecondary institutions; and 

counseling, advising and assistance in the application process for financial aid at 
the postsecondary level. 

Impact on School Counselors 

Clients should report that Talent Search programs: 

provide school counselors with application materials and information; test 
applications; admissions applications; financial aid applications; 

provide school counselors with fee waivers; and 

provide counselors with an information programming source on postsecondary 
and career opportunities by making available Talent Search staff time and 
Talent Search Program resources to the local schools. 

Impact on Postsecondary Admissions and Financial Aid Officials 

Clients should report that Talent Search programs: 

provide admissions officials direct access to potential students: 

serve as a local contact for admissions officials working with local students and 
schools; 
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serve as a conduit for admissions and financial aid materials from 
postsecondary institutions to students and counselors. 

The impacts listed above specify the range of services that are central to clients' needs 
in persisting through high school graduation, for entry to, and, in a more limited way, persistence 
in postsecondary schooling, as well as those sendees that are critical to school counselors and 
college officials in that they are integral to their success in facilitating such outcomes for 
disadvantaged and minority students. Because these impacts are drawn directly from the charge to 
and reported services of Talent Search programs, they are seen as measurable as stated. In 
addition to reporting the actual services received, clients can be expected to give an evaluation of 
those services. The evaluation should center on their perception of the value of those experiences 
and services and an indication of the perceived difference(s) the service(s) made in the 
participants' educational experiences. 

4. Research Design 

If the above conceptualization of Talent Search and its client base is accepted, it is 
then possible to discuss a research design that will enable the United States Department of 
Education to implement a plan of data collection inclusive of that participant base and potentially 
representative. The proposed multimethod design entails secondary data analysis, in-depth 
interview and survey approaches to data collection. 

The purpose and utility of each approach is described below. 

Secondary Data Analysis 

Secondary data analysis of national survey data provides a potentially rich and 
perhaps the only nationally representative sample of participants and non-participants in the 
nation's major funded intervention programs designed to increase access to higher education for 
disadvantaged and or minority students. Such data would allow for important descriptive analyses 
and, to a limited extent, an opportunity to assess the impact of Talent Search services by 
contrasting and comparing students who report being users with those who do not. In addition to 
the general information on completion rates [again, I do not view these as required assessment 
criteria for Talent Search], there would be an opportunity to examine the quality and sufficiency of 
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financial aid between Talent Search participants and non-participants; the range of postsecondary 
choices made by Talent Search participants and non-participants; some limited opportunity to 
compare high school and pre-high school transcripts between Talent Search participants and non- 
participants; and some limited opportunity to examine knowledge about and the importance of 
financial aid for Talent Search participants and non-participants. This of course could be done for 
students matched on a variety of socioeconomic background and school context attributes. The 
potential data sources are listed below. The student-level data are self reported in these national 
surveys and are thus vulnerable to errors of recall as well as reliability and validity. I suggest that 
these threats are real but should not limit the use of these data. Indeed, these data are used 
regularly to address a wide range of educational progress and success questions by scholars in 
several different disciplines. Moreover, prior research has found little evidence of systematic 
response error of bias. 

Beginning in 1972, the U.S. Department of Education initiated a series of national 
longitudinal studies of the nation's schools and students. The high school class of 1972 was the first 
class surveyed, and there are now six follow-up surveys of that group. The next survey was initiated 
in 1980 and included both 1980 sophomores and 1980 seniors. There is now a third follow-up of 
those classes. Finally, there is the national education longitudinal survey of eighth graders in 1988 
(NELS). Each of these surveys contains items requesting that students report their participation in 
each of the TRIO programs, identified as Talent Search, Upward Bound, Special Services and 
Educational Opportunity Centers (3,972 students from the 1972 survey reported having heard of 
Talent Search, while 322 reported having participated in Talent Search. The respective figures for 
the class of 1980 are 2,891 and 489). Moreover, each of the surveys of high school students 
contains supplementary surveys of the student's schools, and in the 1980 survey, parents and 
friends were also surveyed. A rich variety of questions ranging from attitudes and opinions to 
aspirations and expectations are included in these surveys, and many are repeated in each follow- 
up survey. It is important to note, however, that these surveys do not ask students to define what 
they intended by "participated in" or to evaluate the services they received from Talent Search or 
other programs. 
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In-Depth Interviews with Talent Search Clients (Direct and Indirect) 

Franklin (1985) reports that definitions of clients and services are key problems in the 
determination of Talent Search evaluation. One research strategy designed to develop a 
comprehensive understanding of practices and processes of imperfectly understood activities is the 
in-depih interview. Selecting a purposive sample of respondents (program staff and clients) who 
are or have been administrators and/or users of Talent Search services should be the initial step in 
building a refined set of questions for use in later stages of the assessment of the impact of Talent 
Search. Obviously such a sample has to be representative of the variety of social contexts in which 
Taleat Search operates. The interviews should be semi-structured, with greater emphasis placed on 
open-ended items that will elicit the breadth and depth of content needed to yield a sufficient 
range of potential survey items. Again, "clients" here refers to both direct (students and dropouts) 
and indirect clients (parents, counselors, postsecondary officials). The data derived from these in- 
depth interviews should also be regarded as rich qualitative data. A qualitative study in and of 
itself seldom yields results that adequately inform policy makers as to decisions regarding funding, 
but there are few, if any, alternative means for thoroughly identifying and communicating the 
meaning of Talent Search services to individual clients. This is especially so for program services 
that must be geared to the often unique and special conditions under which the service is 
delivered. Finally, it is reasonable to assume that this may be one of the few ways of identifying 
those practices and services that are in some sense standard and essential across programs in 
contrast to those that are unique but nonetheless essential. 

Surveys of Talent Search Participants 

A survey of active Talent Search clients is the desired approach for obtaining a more 
generalizabie base of current data on the impacts listed above. A combination of group- 
administered and mailed surveys is recommended. Group-administered instruments should be 
used with younger respondents and mailed instruments with the older students, except where older 
students who are Talent Search clients are administered services in a group format. In the latter 
instance, a group administration of the survey is preferred. An alternative to the mailed surveys 
would be on-site administration of the survey by non-program staff. The content of these surveys 
should focus on the students' description of services received/anticipated and their evaluation of 
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the quality of the services. Critical emphasis should be placed on the importance (value) of these 
services to the client and the client's perception of her/his alternative sources of similar assistance. 

A survey of indirect clients is also recommended. Indirect clients most probably must 
be surveyed by mail or by telephone, particularly school counselors and college admissions and aid 
officers. Logistically, it is rare that either of the indirect client groups are easily available for a 
group-administered survey. 

Sampling decisions for each aspect of recommended data collection are crucial. The 
purposive sample recommended for the qualitative phase must select respondents who are most 
likely to be best informed about services and activities. This means selecting student clients and 
indirect clients who have in some sense "fully utilized" the project. They are more likely to be 
informative about the scope of activities and services and hence would likely provide a richer body 
of qualitative data. 

The sample design for the survey phase of the research is dependent in large part on 
the operating design of individual projects. For example, the EOC in Washington, D.C uses a 
"club" format for delivery of services to its pre-high school clients, making it unnecessary to sample 
among club participants. A survey instrument could be group-administered where such operating 
procedures exist. Other student clients would have to be identified from project records with 
priority given to current clients. Among these a strategy for selecting a sample from among clients 
with different statuses has to be determined. For example, projects will have currently enrolled 
high school students, currently enrolled college students, dropouts and so forth. Proportionate 
sampling, based on their representation in the total client pool, from among these categories 
should suffice. Surveys of indirect clients would also be highly dependent upon project operations. 
Washington, D. C. has many fewer high schools than Chicago, for example, and this would affect 
the number of school counselors who might be surveyed. Given the total numbers likely to be in 
this category, however, sampling may not be necessary, especially for a mailed or telephone survey. 
I would suggest that the same would be true for college officials. 

Procedurally, I would strongly recommend that the qualitative phase of the research 
be initiated first. I urge this due to the necessity of grounding the development of the service 
indicators to be measured in a clear understanding of operating practices and client experiences. 
Failure to do so might omit critical service activities for some projects while perhaps exaggerating 
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the centrality of others. Moreover, this would seem to facilitate identifying those survey items that 
should appropriately be included for administration to each client category as compared to those 
that should not. This should yield some efficiencies in the collection and management of the 
resulting database(s). 

The above data collection strategies would yield both quantitative descriptive data and 
qualitative data addressing the impact of Talent Search services. The resulting data would offer 
the opportunity for Talent Search to tell its story in a manner satisfactory to congressional funding 
sources and at the same time remain true to Talent Search administrators who are rightfully 
concerned that numbers seldom tell the whole story or even the most important story. 



5* Conclusion 

The proposed study identifying the impacts that could be attributed to Talent Search 
and suggesting a research design for assessing those impacts is intended to be comprehensive and 
thorough. Of course, that will depend upon the funding level and time frame for the research, as 
well as the handling of the qualitative data. It is difficult to envision how an alternative design 
might accomplish the several goals of this assessment. Similarly, it is difficult to advise as to which 
of the proposed components of the research might be sacrificed and still yield a satisfactory 
assessment of Talent Search services. It is my contention that the qualitative phase of the research 
is essential. 

The impacts described in this proposal are viewed as important and are consistent 
with the author's understanding of the scope of services allowable through Talent Search funding. 
There is no attempt herein to expound on each of the listed impacts. This is in part because they 
appear self explanatory and in part because such clarifications will be needed and generated in 
subsequent refinements of this research, especially resulting from the proposed qualitative phase. 
It is likely that different impacts could be identified and some that are listed might be altered or 
even eliminated. 
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Highlights 



A comparison of data on the Talent Search program (TS) from program years 1986- 
87 and 1990-91 reveals that the demographic profile of the Talent Search participant population 
remained largely the same in terms of gender, race, academic and economic status, and kind of 
postsecondary placement. 

In both years the "most typical" Talent Search participant was a white or black female 
in the 12 to 18 age range who was both low-income and a first-generation prospective college- 
attender. She was in grade 9, 10, 11, or 12 and, according to the 1990-91 data, engaged in TS 
activities such as counseling (academic, personal, group or peer), career awareness and 
orientation, and/or financial aid counseling and assistance. If she went on to postsecondary 
education, she most likely attended a public four-year school. 



II. The Data Sources 

This paper summarizes a comparison of the data provided by Talent Search grantees 
on the 1990-91 and 1986-87 Annual Performance Reports. At the end of each program year, each 
grantee is required to submit an annual performance report to the program office at the U.S. 
Department of Education. A blank copy of the relevant pages from the Annual Performance 
Report form that was used for program year 1990-91 appears at the back of this paper. (A blank 
form for program year 1986-87 was not available.) 

As can be seen by looking at the 1990-91 report form, each grantee provides basic 
information on the demographic characteristics of Talent Search participants, the services offered, 
and the number of participants who go on to different kinds of postsecondary education. (The 
qualitative data provided on page 13 of the report form was not analyzed for this paper.) 

For program year 1990-91. data were available for 162 of the 177 grantees who were 
funded. This represents 91.5 percent of the projects. Most of the 15 missing performance reports 
were sent back to the grantees for corrections. After the 15 corrected reports are available, the 
data will be analyzed again. 
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For program year 1986-87, data were available for approximately 68 percent of the 
Talent Search grantees. (The data from that year were taken from a paper that summarized 
performance report data from all of the TRIO programs. The author of that paper did not specify 
the number or percentage of Talent Search grantees represented; he only noted that all of the 
TRIO data combined represented about 68 percent of the TRIO grantees.) 

III. The Quality of the Data 

Each 1990-91 Performance Report was checked for internal consistency and missing 
data. Those reports that had missing data or displayed inconsistent data were sent back to 
grantees. Most grantees made corrections to the reports and sent them back. However, some 
grantees were not able to obtain all missing data. Thus, for certain demographic characteristics 
such as age and academic status, some data are missing. 

As noted above, the data from the 1986-87 Performance Reports were taken from a 
summary report written by an Education Department employee. In the "Methods Summary" 
section of that paper, the author noted that each Performance Report was edited for internal 
consistency and missing data. Reports that contained errors were sent back to the grantees for 
corrections. In many cases a common pattern of mistakes could be identified, and Performance 
Reports that contained such mistakes were corrected by imputing new numbers without contacting 
the grantees. 

IV. Profile of the Talent Search Projects 

A profile of the Talent Search projects is presented below. The size of the projects, 
the type of grantee, and student enrollment at the institution of higher education (IHE) grantees 
were all examined. 
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A. Size of Projects 



The total number of students served by the projects in 1990-91 ranged in number from 
404 to 17,470. However, the project that served 17,470 was an outlier, for the project that had the 
second highest number of students served only 3,723. Therefore, most projects fell into the 404 to 
3,723 range. 

If the project serving 17,470 is included, the average project size was 1,098. If that 
large project is excluded, the average project size was 996. 



B. The Kinds of Grantees 

Just as there was variation in the size of projects, the kinds of grantees varied. In 
1990-91, 123 of the 162 grantees were institutions of higher education and 39 were "other" kinds of 
grantees. (The great majority of these grantees were probably community-based organizations, but 
the kind of "other" grantee was not examined for this paper.) Of the grantees who were 
institutions of higher education, 61 were 4-year public colleges, 23 were 4-year private colleges, 37 
were 2-year public colleges, and 2 were 2-year private colleges. Data on the type of grantee were 
not provided in the paper on the 1986-87 data. 



Table 1 

Distribution of Talent Search Grantees by Type of Grantee 
(Program Year 1990-91) 



Grantee Type 


Number of Grantees 


Percentage of Grantees 


4-Year Public 


61 


38% 


2-Year Public 


37 


23 


4-Year Private 


23 


14 


2- Year Private 


2 


1 


Other 


! 39 

i 


24 


Total 


| 162 


100 
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C. Talent Search Grantees that Were Institutions of Higher Education (IHEs) 

Although the majority of grantees were IHE's, the distribution of IHEs in the Talent 
Search grantee population by type and control did not reflect the distribution of IHEs around the 
United States. A comparison of the distribution of the Talent Search grantees that were 
institutions of higher education (IHEs) with the distribution of all IHEs by type and control, 
reveals that 4-year public schools were overrepresented in the Talent Search grantee population. 

More specifically, while only 17 percent of all IHEs in the United States were 4-year 
public schools in 1989-90, 50 percent of the Talent Search grantees that were IHEs were 4-year 
public schools. Likewise, 4-year private IHEs were underrepresented in the grantee population. 
While 43 percent of all IHEs were 4-year private schools in 1989-90, only 19 percent of the Talent 
Search grantees were 4-year private schools. 



Table 2 

Distribution of Talent Search Grantees and All Institutions 
of Higher Education by Type and Control 



Type of Institution 


Percentage of TS Grantees 


Percentage of Institutions 
in United States 


4- Year Public 


50% 


17% 


2- Year Public 


30 


27 


4-Year Private 


19 


43 


2- Year Private 


2 


12 


Total 


100 


100 



NOTE: The Talent Search grantees represented in this table only include those grantees that were 
Institutions of Higher Education. (The data are for Program Year 1990-91.) The data on all 
institutions in the U.S. are for school year 1989-90. 

SOURCE: The distribution of all Institutions of Higher Education comes from the Digest of 
Education Statistics, 1991 . 



The following tables provide more information on the 1990-91 IKE grantees. As shown in Table 3, 
the four-year public schools were on average larger than the ether grantees in terms of the total 
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number of students enrolled. Table 4 shows that at least 50 percent of the students at 23 percent 
of the IHE grantees were minorities. This is an interesting finding in light of the fact that the 
majority of Talent Search participants were minorities in both 1986-87 and 1990-91. (See the 
discussion of race in section V of this paper.) 



Table 3 

Average College Student Enrollment at 
Talent Search Grantees in the Fall of 1990 



Grantee Type 


Average Enrollment 


4- Year Public 


13,539 


2-Year Public 


5,552 


4- Year Private 


4.361 


2- Year Private 


636 



NOTE: The data on grantees are limited to Institutions of Higher Education. 
SOURCE: Integrated Postsecondary Education Data System (IPEDS), Fall 1990. 
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Table 4 



Distribution of Talent Search Grantees 
by The Percentage of 
Minority Students Enrolled 
(Fall 1990 Data) 



Percentage of Minority 
Students Enrolled At 
Institution 


Number of Institutions 


Percentage of Institutions 


Less than 10% 


35 


28% 


10% - 24.99% 


44 


36 


25% - 49.99% 


16 


13 


50% - 89.99% 


10 


8 


90% or More 


18 


15 


Total 


123 


100 



NOTE: Data on grantees are limited to Institutions of Higher Education. 
SOURCE: Integrated Postsecondary Data System (IPEDS). 



V. Profile of the Talent Search Participants 

The second part of the analysis involved an examination of the demographic 
characteristics of the Talent Search participants, such as income, first-generation-college status, 
race, gender, age, and academic status. 

A. Income and First-Generation Status 

The economic profile of Talent Search participants was virtually the same across the 
two comparison years- 1986-87 and 1990-91. Despite the fact that the law requires that two-thirds 
of Talent Search participants be low-income and first-generation, far more than two-thirds ot the 
participants in both years met this criterion. As shown below. 72 percent of the 1986-87 
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participants and 73 percent of the 1990-91 participants were both low-income and first-generation 
college students. Only 7 percent in 1986-87 and 6 percent in 1990-91 were neither low-income nor 
first generation. 



Table 5 

Distribution of Talent Search Participants by Low-Income and 
First-Generation Status 



Status 


1986-87 Participants 


1990-91 Participants 


Low Income and First 
Generation 


72% 


73% 


Low Income 


7 


6 


First Generation 


15 


15 


Other 


7 


6 


Total (Percent) 


100 


100 


Total (Number) 


137,916 


177.808 



NOTE: Distributions may not total 100.0 percent because of rounding. 



B. Race 

Much like income and first-generation-college status, from 1986-87 to 1990-91 the 
racial profile did not change in any major way. Whites, Blacks and Hispanics made up the largest 
percentage of the Talent Search population in both years. However, two small changes can be 
seen-the percentage of whites decreased by six while the percentage of Hispanics increased by 
four. 
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Table 6 

Distribution of Talent Search Participants by Race 



Race 


iyoo-o7 Muaents 


1QQO.Q1 ^tnHpnrc 

l""Uvl OIUUCIA3 


Black 


32% 




White 


3 / 


^1 

J> 1 


riispanic 


22 


26 


Asian 


4 


5 


Native American 


5 


4 


Other 


N/A 


Less than 1 


Total 


100 


100 


Total (Number) 


137,916 


177,808 



NOTE: Distributions may not total 100.0 percent because of rounding. 



C. Gender 

The gender profile changed even less than the income, first-generation, and racial 
profile. The ratio of females to males was the same in 1986-87 as in 1990-91-58 percent female to 
42 percent male. 



Table 7 

Distribution of Talent Search Participants By Gender 



Gender 


1986-87 Participants 


1990-91 Participants 


Female 


58% 


58% 


Male 


42 


42 


Total (Percent) 


100 


100 


Total (Number) 


13-.916 


| 177,S0S 
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D* Age 



Data on age were only available for the 1990-91 year, so a comparison of the age of 
the population over time was not possible. 

In 1990-91, the vast majority of Talent Search participants-83 percent, or 146,984- 
were between the ages of 12 and 18, and only one grantee did not serve students in that age range. 
Thirteen percent (22,825), were between the ages of 19 and 27, and 4 percent (7287) were 28 years 
of age or older. Talent Search projects are legally allowed to serve persons over 27 years of age 
when there is no Educational Opportunity Center in the same target area as the Talent Search 
project. 

Table 8 

Distribution of Talent Search Participants By Age 
(Program Year 1990-91) 



Age 


Number of Participants 


Percentage of Participants 


12 to 18 


146,984 


83% 


19 to 27 


22.825 


13 


28 and older 


7.2S7 


4 


Total 


177,096 


100 



NOTE: The total for this distribution is less than 177,808 due to missing data on age. 
E. Academic Status 

Like most of the other demographic characteristics, the academic profile of the TS 
population was similar in 1986-87 and 1990-91. In both years over three-quarters of the 
participants were in high school-79 percent in 1986-S7 and 75 percent in 1990-91. (For the 1990- 
91 data, we assumed that 9th graders were in high school.) 
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Among those participants not enrolled in middle school or high school in both years, 
the largest number were high school graduates with no postsecondary education. (Note that the 
data for the two program years have not been combined into one table because some of the 
categories were not comparable.) 

Table 9 

Distribution of Talent Search Participants by 
Academic Status 
(Program Year 1990-91) 



Academic Status 


Number of 
Participants 


Percentage of 
Participants 


Grades 7-8 


19.426 


11% 


Grades 9-11 


65.288 


38 


Grade 12 


64.053 


37 


High School Dropout 


6.272 


4 


High School Graduate with 
no postsecondary 


12.540 


7 


Postsecondary Dropout 


3.190 


2 


Talent Search Dropout 


1.524 


1 


Unknown - "Unable to 
Track- 


4.235 


2 


Total 


173.651 


100 



NOTE: Distributions may not add to 100.0 percent due to rounding. The total for this distribution 
is less than 177.808 due to missing data on academic status. (Beginning in fiscal year 1993, services 
will be provided to 6th graders as a result of 1992 Amendments to the Higher Education Act.) 
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Table 10 



Distribution of Talent Search Participants by 
Academic Status 
(Program Year 1986-87) 



Academic Status 


Number of Participants 


Percentage of Participants 


Enrolled in high school 


109,134 


79% 


High school dropouts 


9,193 


7 


High school graduate with 
no postsecondary 


14,318 


10 


Postsecondary dropout 


5,271 


4 


Total 


137.916 


100 



VI. Profile of Talent Search Activities 

A comparison of Talent Search activities across years was not possible because the 
paper on the 1986-87 data did not discuss activities. 

In 1990-91, no one activity or service was offered by all grantees and there was no 
single service that every TS participant received. As can be seen in Table 11, the service most TS 
participants received was counseling. Fully 78 percent (135,614) of the TS participant population 
received some kind of counseling-academic, personal, group, or peer. In addition, Table 12 
reveals that 96 percent of the projects (156) offered some kind of counseling. 
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Table 11 



Activities/Services Offered by Talent 
Search Grantees in Program Year 1990-91 by 
the Number and Percentage of Participants 



Activity 


Number of Participants 


Percentage of Participants 


Counseling (Academic, 
Personal, Group, Peer) 


1*55 614 


78% 


Career Awareness and 
Orientation 


114.333 


66 


rinancial Aid counseling 
and Assistance 


101,033 


58 


Campus Orientation and 
Visitation 


31587 


19 


cultural /vciiviLics 


16,424 


9 


Tutorial Assistance 


14,727 


8 


otudy oKins 


7.950 


5 


compuicr-/\ssisicu 
Instruction 


5.652 


3 


GED Training 


1.857 


1 


Other (Instruction in subject 
other than English, reading, 
writing, math) 


1,556 


1 


Math 


1,258 


1 


Writing 


921 


1 


Reading 


840 


Less than 1 


English 


692 


Less than 1 



Career awareness and orientation, and financial aid counseling were the other 
services that most participants received and that most projects offered. Campus orientation and 
visitation was offered by 85 percent of the projects but only 19 percent of the participants 
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participated in this activity. This discrepancy may be due to the possibility that only 12th graders 
participated in the campus activities. 

It is also striking that 57 percent of the grantees offered tutorial assistance but only 8 
percent of Talent Search participants received such assistance. This may reflect the possibility that 
many TS participants received tutoring services from other sources or were not judged to need 
such services even though such services were available at many TS projects. 
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TABLE 12 



Activities/Services Offered by Talent Search Grantees 
in Program Year 1990-91 by the Number and 
Percentage of Grantees Who Offered Each 
Activity/Service 



Activity 


Number of Grantees 


Percentage of Grantees 


Financial Aid Counseling 


158 


98% 


Counseling (Academic, 
Personal Group, Peer) 


156 


96 


Career Awareness and 
Orientation 


156 


96 


Campus Orientation and 
Visitation 


138 


85 


Tutorial Assistance 


93 


57 


Cultural Activities 


66 


41 


Computer-assisted 
Instruction 




70 


GED Training 


32 


20 


Study Skills 


25 


25 


Reading 


9 


6 


Math 


9 


6 


Other (other academic 
subjects) 


9 


6 


Writing 


8 


5 


English/English Proficiency 


6 


4 
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VIL Postsecondary Placements 



The placement of Talent Search participants in postsecondary educational institutions 
was analyzed in terms of the percentage of participants who went on for postsecondary education 
and the kind of postsecondary institution attended. 

A. The Percentage of Participants Who Went On for Postsecondary Education 

It is impossible to know the percentage of the participants who were eligible or ready 
for postsecondary education who in fact went on for such education after participating in Talent 
Search in 1986-87 or 1990-91. This is because there were no data on the number of participants 
who were "eligible" for postsecondary education. Therefore, in order to estimate the percentage 
who went on for postsecondary education among those who were eligible, certain assumptions had 
to be made about eligibility. 

We assumed that those who were eligible were the following groups of participants: 
12th graders, high school graduates who did not have any postsecondary education, and dropouts 
from postsecondary institutions. Because we could not identify 12th graders for Program Year 
1986-87, we could not estimate the percentage of "postsecondary eligibles" who went on for 
postsecondary education. 

However, for the 1990-91 program year, these groups were easy to identify, and 
contained 79,783 participants in all. Since the total number of 1990-91 participants who went on 
for postsecondary education was 5S.255, this represents 73 percent of those who were eligible. 

B. The Kinds of Postsecondary Placements 

The percentage of TS participants who attended each kind of postsecondary 
institution hardly changed from 19S6-87 to 1990-91. In both years the largest percentage of 
students went on to public 4 a ear schools and the second largest percentage of students went on to 
public 2-year schools. fSeven:\-::\e percent in 1986-87 and 77 percent in 1990-9: '.vent on to public 
schools.) 
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Likewise, approximately the same percentage of students went on to 4-year private 
schools in both years-9 percent in 1986-87 and 10 percent in 1990-91. 

Because the data on postsecondary placements were not sorted into all of the same 
categories across the two program years, we cannot compare the percentage of participants who 
went on to the army, proprietary schools, or vocational/ technical schools. Only the 1990-91 
reports collected data for those kinds of placements. 

At the end of that program year, six percent, or 3,425 participants, attended 
vocational schools. It is possible that this underrepresents the number of students who went on to 
vocational programs, since some of the students who went to two-year public and private colleges 
may have enrolled in vocational or technical programs. It is also important to note that at the end 
of the 1990-91 year, the armed forces received about the same number of TS participants as did 
private, non-profit colleges and proprietary schools. 
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Table 13 



Distribution of Talent Search Participants Who Went On for 
Postsecondary Education by Kind of Postsecondary 
Placement 
(Program Year 1990-91) 



Kind of Placement 


Number of Participants 


Percentage of Participants 


Public 4-year 


23,416 


40% 


Public 2-year 


21,809 


37 


Private, nonprofit 4-year 


5,935 


10 


Nonprofit 

vocational/technical 


3,425 


6 


Army 


1,456 


3 


Private, nonprofit 2-year 


1,111 


2 


Proprietary 


1.103 


2 


Total 


58,255 


100 



NOTE: The "Percentage of Participants" in this table represents the percentage of students who 
went on to each kind of institution as a percentage of those students who went on for 
postsecondary schooling (58,255). 
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Table 14 

Distribution of Talent Search Participants Who Went On for 
Postsecondary Education by Kind of 
Postsecondary Placement 
(Program Year 1986-87) 



Kind of Placement 


Number of Participants 


Percentage of Participants 


Public 4-year 


20,701 


40% 


Public 2-year 


17,708 


35 


Private 4-year 


4,481 


9 


Private 2-year 


1,372 


3 


Other 


6,918 


14 


Total 


51,180 


100 



NOTE: Distributions may not total 100.0 percent because of rounding. 



VIIL Conclusion 

A comparison of the 1986-87 and the 1990-91 data reveals that the economic, racial 
gender, and academic profile of the Talent Search participant population was similar across those 
program years. In both years the majority of the participants were low-income and first -generation 
females, were Black and/or white, and were enrolled in high school. 

A comparison also reveals that the majority of participants who went on for 
postsecondary education went to public institutions in both years. 

A comparison of the activities/services offered by Talent Search grantees could not be 
made since such information was not available from 1986-87. However, in Program Year 1990-91, 
the most common activities were counseling (academic, personal, group, peer), career awareness 
and orientation, financial aid counseling, and campus orientation and visitation. 
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SECTION II - INFORMATION ON PROJECT PARTICIPANTS 
A, Numoer ot Participants Assisteo During the BuQQet Penpj: 



ri ^ Participant DistnDution on the Bas is o t EMtaiDility CriU fi a 

I 

" ELIGIBILITY CRITERIA j NUMBER OF PARTICIPANTS 

j 

1. Low-income anc First-generation i , — 

i 

2 . Low- 1 ncon>e only i . 

i 



3 . Pi rst-generat ion only I 

j 

4. Other i 

j 

Total (Sum ot lines 1 tnrouoh 4 j 
must aaree with tota i in a aoove ; i 



C. Participant Distnoution py etnmc 6<a^^ioung 

i 

ETHNIC BACKGROUND [NuMBE* v> pa* ; ; W I FA<< : c 

! 

1. American Ina i an / A iasr an N^u ve ■ 

i > 

2. Asian/PaciT ic Islander^ i 

i 

3. Black (Otner tnan Hispanic ■ » 

i 



a . hi span i c • 

5. wmte ( OLner tnan Hiso*mci « 

i 

6. TOTAL (Sum or lines 1 throuan 5 i 
must agree with tota i in nne Aj j 



0, Participant Distripution &y ue n oe r 



I 



NUMBEP OF PAPTICIPA: 



Ma i e 
Fema i e 

TOTAL (Sum or lines 1 ana . 
must aaree * i tn tota i m a 
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c o*rr.ip^nant niRtnbution py Age 




NUMBER OF PARTICIPANTS 


TALENT SEARCH/EOC 




1 . -Ages 12-18 





2* Ages 19-27 

3* Ages 28 and APove 

4. Total sum of l>nes 1-3 
must agree with total in 
A above ) 

UPWARD BOUND 

1 # Ages 13-19 

2* Ages 20 ana Above 

3. Total (Sum ot nnes 1-2 
must agree witn ta i 
in A above. 
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SECTION III - PROVISION OF SERVICES 

Indicate the number ot participants who nave receiver trie 
following Rinds of services during the reporting period: 

Numoer ot 

Service Participant 

Instruction (UB only) 

English/English Proficiency 

Reading . 

Writing 

Study SK i 1 Is 

Mathemat i cs 

Other iSoeciry course offerings j 

GED Training 

Financial Aid Counseling ana 

Assistance (TS, EOC, UB ) 

Counseling lacaaemic, personal, orouc, peer * 

Computer-Assisted Instruction 

Tutorial Assistance 

Cultural Activities 

Career Awareness and Orientation 

Campus Orientation ana visitation 

Dissemination or admission ana financial aid 

information (Educational Opportunity Centers J 



ED ^orm £, 4G-*2r. lu/ei? 



ERLC 



OMB Aoproval Numoer: i»A0-0£>bi 
Expiration Date: 12/31/92 



SECTION IV -PROJECT PERFORMANCE OUTCOMES 
TALENT SEARCH 

A. Participants tnrolUd in project during trie program year 

Numpej. 



B. 



1 . Graces 7-tt 

2 . Grades 9-1 1 

3. 12th graders 

4. . H i gh schoo 1 dropouts 

5. High school graduates with no 
postsecondary education 

6. Dropouts from postseconaar y 
institutions 

7. Dropouts from project 

8. Participants unaDle to track 
Postsecondary Placements 



Numoer 



1. Participants who enrolled in 

postsecondary education since start 
ot program year 

o For those participants reportea 

in B.i above* indicate tne number who 
are enrol led in : 



a. 
D . 

c . 

a. 



t . 

a . 



Public two-year institutions 
Private* non-profit two-year 

insti tutions 
Public tour-year institutions 
Private, non-profit tour 

year institutions 
Public or non-profit vocational t 

tecnmca ! insti tutions 
Proprietary schools 
Arn»ec Forces 



1 1 - 
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SECTION V - SUMMARY OF OTHER PROJECT ACCOMPLISHMENTS 

State briefly other goals and objectives wmch are not covered in 
previous sections of this report ana pnerly describe tne 
project's accomplishments During the raportmg period. Report 
those activities that specifically relate to the Prior Experience 
criteria- and the degree to which you met those oojecT, > ves . It 
necessary, use aoditional pages for tins section. 
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1 J- 



BEST COPY AVAILABLE 



172 



